Split Testing Software — What You Need To Know.

I read once about a man who tripled his conversion rate by changing a single letter in his headline. While that kind of improvement is not common (not by a long stretch), you can quite probably double your conversion rate after testing 10 or 20 items on your page. With that said, there are many things to consider when getting started with split testing.

I can’t discuss every feature of every testing tool out there, but I can give you a major overview of the different kinds of split testing software.

The first main division is between simple split testing (single variable) and multivariate testing.

Single variable split testing is very simple. You just try stuff and measure the results. And all you need is a tool that rotates and keeps score.

There are a few simple split testing tools available. Some are free and some cost money. Google offers a free tool, though you do have to have an Adwords, and an Analytics account to use it.

But simple split testing can be discouraging. When you test a new idea against your control version, sometimes the control version wins. In fact, it often wins. You can get a string of disappointing tests all in a row. What would it be like if you could test 10 things at once? Well, you’d be bound to find at least one or two ideas that led to improvement. And what if you could test all those ideas simultaneously with the same amount of traffic it takes to test one idea with simple split testing? Well, it sounds like magic, but that’s exactly what multivariate testing can do for you.

So there’s the main distinction between single variable testing and multivariate testing. But once you get into the realm of multivariate testing the options seem to multiply again. One difference among multivariate tools is between tools that are hosted on someone else’s server, and ones that you install on the same server as your webpages. Hosted solutions can be slower, and more expensive, but you don’t have to install any software.

The Split Test Accelerator is a good choice if you want to install the software on your server. Kaizentrack is another optoin. If you want a hosted solution, you can look at Sitespect and Vertster.

Taguchi testing is only one kind of multivariate testing. There are also “full factorial” tools, and random independent rotators. When you run a Taguchi test, you run only a small fraction of the possible combinations of your factor levels. You might, for instance, run only 18 pages to test 4374 combinations. This is made possible by orthongonal arrays. This allows you to find the best combination with out running through all of them. You do have to be careful about your factors interacting negatively with each other, but if you take care as you design your test, you can avoid the bulk of the problem with these. The Split Test Accelerator, Vertster, and Kaizentrack are Taguchi testing tools.

When you run full factorial tests, as you would with Google’s Website Optimizer, you create a few factors, and a few options for each factor, and the software cycles through each combination. So, if you have 5 factors with 2 options each, you would cycle through 32 different pages. FFT can be used like a Taguchi test for small test designs. Or, if you run a lot more traffic to it, you can deal even more effectively with interaction effects by just letting all the interactions (good and bad) play out. Note that you can’t run as many combinations with full factorial tools as you can with a Taguchi too.

The Random independent rotators are a third option. These treat factors independently. So you can just plug in a new factor with as many options as you want at any time, and ignore the fact that other factors are being tested at the same time. This is the most flexible form of testing, but it is also plagued with the most interaction effects.

That sums up some of the main difference among split testing tools. There are many other features to compare, but this should give you the big picture.

Dominic DeLong has used taguchi testing tools for years to improve landing pages.