The day after my presentation on multivariate testing at the Geek Dinner, I kept thinking about how I was so astonished that almost no one there knew about or worked with multivariate testing. Perhaps I am being a bit self-deprecating, but I keep thinking that everyone else knows more than I do; so it's strange (and enlightening) that there is a field where I can educate. I wanted to give a quick down down of what I talked about, and hopefully get a bit more information out there.

A/B Testing; smaller and easier

A/B Testing (or Split Tests) is the more well-known, easier to understand cousin of MVT. Basically, take your website, and create an alternate version--different design, different copy, anything you like--and set up your server to send 50% of your visitors to one version and 50% of your visitors to another. Then, you use your basic analytics package (like Google Analytics) or a click-tracking software (like Clicktracks), and determine which version of your website performs better depending on a certain metric (like how many visitors on each filled out a "Request Info" form).

Enter Multivariate Testing, A/B's intimidating but really awesome cousin

The simplest difference between A/B and MVT is the number of "versions" you have on your website. In MVT, generally you set up a number of "factors" of things you will change. These factors could be:

• Link Color
• Header Image
• Introductory Text
• Background color/pattern
• Button color
• Button Text
• Logo Location
• Form Location
• Number of Form Fields
• So on and so forth...

Variations of these above factors, for something like "Link Color", would be "Blue Link Color" and "Green Link Color". A MVT is simply taking of all these factors and variations to create every single combination and finding out which combination of everything would create the best possible version of your website.

Full Factorial vs. Fractional Factorial

Obviously, the more factors and variations you combine, the larger exponentially the number of versions to test would become. A full factorial is simply every single version is tested, and a fractional factorial is only a subset of the number of variations--testing a random fourth of the entire number or so. This isn't going to give perfect results, but my particular package will analyze how each of the groups perform, and determines which factors give the most positive impact and which give the most negative impact. So if you have a ton of ideas and things to test, you can use a fractional factorial to weed out the bad and negative ideas, then run a full factorial to find the best combination of the winning variations.

More information to come...

Hopefully the above makes sense; I'm going to write more about MVT testing soon but this post should give a good, quick overview of what A/B and Multivariate testing are and what they can do. If you have any questions, let me know in the comments!