The Best CRO Tool to Boost Your Profit: Split Testing vs Website Personalization

The Best CRO Tool to Boost Your Profit: Split Testing vs Website Personalization

  • A/B testing is still king. 67% of all companies still list it as their top favourite CRO tool.
  • Personalization is coming up strong. 91% of marketers are already using it, or plan on making it a priority in the near future.
  • Personalization is the proven strategy for increasing profits via retargeting strategy. Meanwhile, split testing a mere headline can drive sales upward by 89%!

Some experts in conversion rate optimization (CRO, in short) will have you believe that the age of website personalization is upon us. Conversely, they say, A/B testing, or split testing as it is otherwise known, is all but dead. Is this true, then? In a sea of experts, how do you know which ones to give credit to? As always, you look at the facts and figures and see who comes closer to the truth.

How many marketers use each of the 2 CRO methods?

If we go by facts and figures alone, the latest data pooled by the 6th yearly Conversion Rate Optimization Report states quite the opposite. According to the survey cited above, 67% of all the companies asked list split testing as their CRO tactic of choice. What’s more, this is no fad, but a well-established trend. A/B testing has topped the CRO methods charts for four years in a row.

At the same time, website personalization is not doing half bad either. It may have started out as the underdog in this head-to-head race, but an equally the 2014 State of Online Testing report revealed that 59% of all online marketers are currently using some form of target audience segmentation. Even more importantly maybe, 32% also make use of dynamic content, in their endeavours to personalize their website visitors’ online shopping experience.

2014 poll results on how many internet marketers personalize their websites

2014 poll results, indicating the relatively low use of website personalization across the marketing industry

Image source:

Actually, if you think about it, personalization is by no means anything new under the sun. It’s been around for as long as abandoned shopping cart emails and on-site messages. That specific retargeting method has consistently proven successful and it is, after all is said and done, based on targeting a specific subset of site visitors. Conclusion: personalization may not reign supreme (yet), but it’s definitely gaining traction at a rapid pace.

So, then, could it be true that personalization will soon oust A/B testing? Should you do away with the latter altogether, while focusing all your CRO efforts on the former?

The short answer: no.

To elaborate, comparing A/B testing with website personalization has its limits. Of course, they both aim to up the number of converting leads, but they each boil down to significantly different things. To paint a clear picture for you, we’ve lined up these two CRO methods side by side.

Website personalization versus A/B testing: Pros & Cons

Personalization is a complex effort

While the above sentence may strike you as a banal truism, it does hold water. Try to personalize your site on a whim and you’re likely to end up with disastrous results. Instead, take baby steps toward this goal; don’t dive head-first into the complicated world of dynamic content insertion. Here are the first steps that any personalization plan should be based on:

  1. Identify the main traffic segments. In order to do this, you’re going to want to look at some specific metrics:
    1. New visitors & Returning traffic
    2. Devices used
    3. Visitor countries
  2. Understand user behaviour for each segment. This involves more data analysis:
    1. Page depth – how deep into your website do specific subsets of visitors go? Or, in plain English, how many consecutive links do they click on?
    2. Visit time – how much time does each audience subset spend on-site, on average?
    3. Scrolling – how do fast do the users gloss over each page on your site that they visit?

Look at the resulting figures and percentages and ask yourself this: “based on user interaction with my page, how can I improve UX for each subset of visitors?” Your main goals should be to keep them on the site for longer, get them to meaningfully interact with it, and eventually have them convert.

  1. Find the pages worth personalizing. This is probably the most delicate aspect of onsite personalization, since there’s no such thing as a foul-proof method to it. Your best bet is to keep rummaging through your analytics backlogs. Sooner or later you’re bound to find pages that are under-performing and that you might consider personalizing for better results. Do bear in mind, however, that not all pages are worth either A/B testing or personalizing.

You need significant traffic for relevant A/B testing

All too often, would-be marketers get confused when it comes to understanding how split testing works. Most of them do know that sample size matters and that, by and large, the bigger is usually the better. But then they fall for some testing tool which promises 95% statistical significance. The problem with this concept is that it’s mutable—you can even have 99% significance for a specific test result one day, only to find that the situation has turned by 180 degrees only a few days later.

In fact, that’s exactly the point of this great analysis by Peep Laja. The takeaway is to run tests for as long as possible, provided that you don’t actually end up losing traffic over them. But don’t settle for a small sample size and end the test early: in the longer run, you might come to realize that you implemented a change that yielded no results. Or, worse even, you went in for one that negatively affected your sales.

screenshot of optimizely results in implementing two split test variants over time

Laja tested for a change that initially lost out by 89.5% to the control. A few days later, the test was looking 95% favourable for the treatment. Eventually, as the results from Optimizely show, after four weeks of testing, there is no difference to speak of, between test and control.

Image source:

Split testing is only for verification

Not to undermine its importance, but split testing does not equal CRO, as some may be tempted to believe. As a matter of fact, there are a lot of techniques that fall under the umbrella of conversion rate optimization. You can find an extensive, yet non-exhaustive list right below. It’s a visual outtake from a 2011 Conversion Rate Optimization Report, which basically showed that businesses are concerned with CRO and try to deploy as many strategies as their time and money allow them.

online optimization poll results showing numerous methods of CRO

Image source:

In the CRO ecosystem, split testing will only take care of one thing: making sure that the hypotheses you’re willing to improve your website on are valid. It pits one speculation against a control and reveals which variant is the winning ticket. In order to get it right, though, you’ll need to elaborate your testing hypotheses correctly. Armed with that kind of expertise, there’s little you can’t find about optimizing your sales funnel. We’ll just let the results speak for themselves:

movexa landing page, before and after CRO following split testing

Image source:

A/B testing or website personalization? Both!

Yes, you read that correctly. Split testing is not going anywhere, even though 91% of marketers polled in a recent survey by Evergage said they’re already using personalization, or are planning to make it a priority in the near future. In fact, the best approach is to use both strategies complementarily. Personalization will go a long way in your overall CRO efforts; however, some audience segments will be too small to split test. Yet you can split test on similar, but less refined sets (i.e. larger sets) beforehand. Try to apply the same results to the segments you can’t test and see how your personalization efforts fare, in terms of improving optimizations. At the end of the day, the key is to know when you need to and can split test, and when other CRO tools, like optimization, will work better.

We will be happy to see your thoughts

Leave a reply

Login/Register access is temporary disabled
Compare items () compare