Much has been said about A/B testing techniques and the value that Website Testing can bring to website owners. In this article, I will not discuss what to test or how to do it, but how to analyze what you have already done.

I have written in the past about the differences between A/B and Multivariate Testing and how to choose between them. Back then, I wrote that one of the advantages of multivariate tests is that you can analyze each and every interaction between elements in a much deeper level.

However, one of the points that I missed is that A/B Tests are easier to be analyzed using Web Analytics tools. Since every test version is a page, they are being tracked by default in your analytics reports. This is not true for Multivariate Tests since the versions are combinations of multiple elements, which are not served as a page but brought into the page by external tools.

In this article, I will go over a technique that can be used in order to make the most out of your Google Analytics implementation so that you can understand the bigger picture of your test (independently of which testing tool you are using).

Analyzing A/B Tests Results Beyond Success & Failure Metrics

Usually, when performing an A/B Test, marketers implement the codes on the original page, on the test page and on the conversion page. This means that the testing tool will track the number of visitors to each of these pages and report back the percentages of visitors viewing each page and their conversion rates. This approach misses two important issues:

  1. Beyond Success And Failure Metrics: some websites (mainly ecommerce) may have completely different values to conversions, meaning that tracking success and failure as a binary decision might be misleading. As George Orwell wrote on the Animal Farm: “All animals are equal, but some animals are more equal than others.”
  2. Engagement Metrics: sometimes the objective of the test may not be to reach a goal page, but to decrease the bounce rate, to increase time on page, or pages per visit.

Since very often the information above is already available in Web Analytics tools, it is logical to integrate testing results into them. So, if you are using Google Website Optimizer, Visual Website Optimizer or other tool of your choice, here is a way to improve the measurability of your tests using Google Analytics.

Setting Up Google Analytics to Track A/B Test Properly

Google Analytics content reports do not show a tab for goal conversion per page. That is understandable, since it is not possible to attribute conversions to specific content. Google’s solution was to create the $ index. Here is this metric’s description as provided by Google:

$ Index is the average value for a page that a user visited before landing on the goal page or completing an Ecommerce transaction (or both). This value is intended to give you an idea of which page in your site contributed more to your site’s revenue. If the page wasn’t involved in an ecommerce transaction for your website in any way, then the $ Index for that page will be $0 since the page was never visited in a session where a transaction occurred.

Since we are discussing A/B tests, by default you will be able to analyze the $ index of each of your test variations (if you have ecommerce this will be defined according to it; if not, be sure to set your goal values when setting your goals on Google Analytics).

But if you want more detail to be collected regarding your test variations, you can also use Custom Variables. Basically, to set a custom variable is to add a value to the user cookie that will persist across the life of the visitor’s cookie. This means that this additional segmentation capability will allow us to compare between pages, both when it comes to engagement metrics and when it comes to conversion metrics.

Here is a sample of the Google Analytics code that should be added to each test page:

body onload="_gaq.push(['_setCustomVar', 1, 'abtest', 'variation2', 1]);"

On the example above, we see the following:

  1. The custom variable is set to slot 1 (out of 5): this important as you might overwrite other variables if they share the same slot.
  2. The name of the variable is “abtest”
  3. The value is “variation2″: each variation should have its own value. If possible, there should be an explanatory name in order to facilitate analysis, e.g. “big_picture”, “long_form” or “wonderful_graph”.
  4. The scope is 1 (visitor level): this should be set to 1 as you want your returning visitors to still have the variable attached to them. If you use 2 or 3 that won’t happen.

Below is a sample of which metrics you will be able to use to analyze your A/B Tests.

Custom Variables & Metrics within Google Analytics

Integrated A/B Tests

As Michal Nassimian wrote on A/B Testing for SEO: “Search Marketing should be viewed as a conglomerate of variables, where SEO, paid campaigns and Web Analytics are intertwined. Each effort to optimize a website should strengthen all others to increase synergy and boost revenues.”

I believe that by integrating Tests into Analytics tools we will be one step closer to an integrated optimization strategy.

Related Topics: Advanced | Channel: Analytics | Google: Analytics | Google: Website Optimizer | How To: Analytics | Search & Analytics


About The Author: is the Founder of Conversion Journey, a Google Analytics Certified Partner. He is also the founder of Online Behavior, a Marketing Measurement & Optimization website. You can follow him on Google+ or Twitter.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Marco La Cava

    Great Article!
    Instead of using Custom Variables, do you think should be possible to track the pages, naming them as Virtual Pageviews:
    _gaq.push(['_trackPageview', '/HomeControl.html']);
    _gaq.push(['_trackPageview', '/HomeVersion1.html']);
    _gaq.push(['_trackPageview', '/HomeVersion2.html']);

    Naturally, the web user will see only the page home.html.

    I was wondering that in this case should be possibile to create different goals/funnels for each different version og the pages.

    Should it works?



Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide