I have had the experience of working on or with numerous sites that had high-end analytics packages like WebTrends or Omniture, but they were doing little or no split testing. The cost of having these packages is only validated if you are getting actionable data from them, and you’re making or saving your organization money with that data! Magento, an inexpensive ecom platform, is now integrated with a free optimization (and analytics) platform, enabling customers like Wearport.com to make data-driven decisions and optimize the visitor experience by answering questions like “would it be better if we actually displayed the prices of these items?”
Moving along, the obvious exec is question “how much money can you save or make through optimization?” Well, let’s look at an example. I was working on a client’s site, doing some basic analytics consulting to repair some problems, and I suggested an overhaul of their product page. There were some best practices that they needed to put in place, and I had some hunches about some aspects of the layout that I thought would improve their conversion rate. Primarily this had to do with the add-to-cart button and the options (usually size and color) selection. I knew we could run some simple split tests on the layout and options, and find the one that resulted in the best conversion rate. I was fairly certain I could raise the add-to-cart rate by 5%.
At first they balked at the idea because of the costs, of course. But I broke it down for them: $20 million in revenue from your website this year, what if you could boost that 5%? Another $1 million in revenue sounded pretty good. So, using Omniture SiteCatalyst, without any of the additional Omniture tools like Test&Target, we set up an A/B test. The developer I was working with was quick and talented, making things much easier. The developer wrote a quick script to randomly display visitors either the old version of the product page, or a new version with a cleaner, more strategically placed add-to-cart button. Tracking the version in our own cookie, we ensured they’d see the same version throughout their visit, and on subsequent visits. We set up an custom conversion event in SiteCatalyst. After two weeks, we analyzed the results. We saw a 10% boost in cart-adds, which resulted in an almost 10% boost in orders!
Next, we tackled the options. Usually color and size were the two options, and they were separate drop downs; one drop down to select color and another to select size. An obvious problem with this is knowing the availability of a size/color combo. We combined the drop downs, so colors were now listed along sizes (e.g. Grey 36, Grey 32, etc.). After two weeks, a 4% increase in cart adds, and a 5% increase in revenue!
Needless to say, the client became a strong believer in split testing. There were of course development and consulting costs involved in this. But an overall 14% boost in sales more than covered those costs, not to mention the long-term value of acquiring new customers by improving the visitor experience.
So, where do you start with split or multivariate testing? Anywhere! Start small, make one educated and solid assumption, and test it out. Watch the results, and then move on to another change, or, if you were unsuccessful, modify your first assumption and try again. The critical objective is to begin to make testing part of your decision making process. Next time you are sitting with your developer, marketing personnel, or site manager, deciding what you think “looks”, “would work”, or “worked at company x” better, just remember that your visitors will do a much better job of letting you know what works.
