How Amazon Does A/B Testing (via the Winery Web Site Report et al.)
Posted on February 26th, 2007 in Business, Design, Product, Technology |
One of Brad’s posts led me to The Winery Website Report Blog which so far has been fantastic. The initial post was about “failing to address a point of pain” in talking about the ‘hiatus’ of the Winery Report. It led me to a post titled “What should you put on your home page?” In it, there’s a link to a presentation (pdf - 2Mb) at eMetrics 2004 about Amazon and how they use automation and A/B testing to decide what goes on the home page. One of the coolest things about this is that it goes through some of their iterations and shows how data is often counter to intuition.
The bottom line - measure, measure, measure and let the best performing idea win. It’s hard to commit to this concept because designers, executives, product managers, all have their own ideas about what will and won’t work. The beauty of a consumer website is that if you’re willing to put yourself in a position to iterate efficiently, you can quickly find strategies that work.
Related Posts:
Adam Duvander - Simplicity Rules - Data Rules Amazon - Simpler Home Page Design Leads to Drop in Orders
Greg - Linden - Geeking with Greg - Talk on Amazon A/B Testing
3 Responses
Thanks for the nice words, Rahul. Much appreciated.
My pleasure, Mike. Thanks for stopping by.
[...] Measuring consumer behavior and watching consumers interact with your site are the best ways to find out what works and what doesn’t. There’s a great post on Signal vs. Noise about some of the things behind Amazon’s success. Two paragraphs that jumped out at me: Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests… [...]