AB Testing - A Crash Coarse in Concept

Years ago we hired our first pedigreed marketing director and she started us down the path of split testing.

If you aren't familiar with split testing, it's basically a method by which two or more "things" can be tested against each other simultaneously.

Why is simultaneity significant? Well, consider an alternative in which you test the first thing for a week, and then you test the second thing during the next week. What if people you were testing in the first week had just received their paychecks and had money burning a hole in their pockets while in the second week, the second thing didn't experience similar luck? You might draw the conclusion that the first thing was preferred by the audience while the second wasn't. And, that would be a mistake folks that don't split test might easily make--without even realizing the data was sour.

Using thing one and thing two in a simultaneous split test theoretically overcomes many obstacles. If you imagine showing five customers the first item and another five customers the second item randomly within a short period of time then you have a better chance of observing the preferences related to the things rather than external factors.

There are a number of issues to overcome in running such a test and I'll cover those in greater depth as this project continues...

Interested in more info? Give feedback.