There comes a time in life when we decide that we know what we know and what we don’t, well its to late to look it up anyway. That used to be the case with me and website optimization. Over the years I’ve heard words such as A/B testing and multi-armed bandit thrown around by more knowledgeable friends and always wondered what they meant. We had Hal Varian speak to us at Startuponomics last year and mention how the multi-armed bandit was standard operating procedure at Google in terms of how they optimized their sites.
In the recent past, a minor storm has been brewing in this teacup among practitioners of both these mechanisms in terms of what works well and under what circumstances. As the geeks in the reader list might be aware, The Obama campaign has been A/B testing their website and email campaigns too, to get better conversion rates. For a quick summary of references and arguments on both sides, you can read this blog post. If you like to watch and listen to talks rather than read websites, you can watch this talk around recommender systems and optimization methods such as MAB to make them better. Coincidentally enough, one of my wiser and smarter colleagues was telling me about the similarities of the epsilon-greedy MAB algorithms to the adaptive filter coefficient optimization and other steepest gradient algorithms in digital signal processing.
In short, the argument seems to be that if you are a startup that is optimizing over a small set of visits, then the multi-armed bandit approach may not give you statistically significant results and you might in fact be optimizing away from your best solution. But if your changes can be done incrementally, and your visit volume is large enough, the MAB approach allows you to track towards the more optimal options quicker while only running experiments on a much smaller subset of your audience. I won’t pretend to know more than this on the topic, but for all my friends that are starting companies where conversion rates are important, this is an issue that you need to look into and understand as you grow. With tools such as the Google Website Optimizer and companies such as Optimizely, it has gotten much easier to do some of this as well.
The key is to build testability into the product, especially one that interacts with a large chunk of its customer base through a website. It can be said with a fair amount of certainty that what we might assume to be the optimal layout, choice of colors, positions of buttons etc might turn out to be not as optimal as other options. For startups that do not have the money to conduct extensive market research surveys, A/B testing or multivariate testing provide a quick way to improve conversions and CTRs through trial and error. If you know more about this, I would love to buy you lunch and pick your brain sometime, so let me know!