A/B Testing – Part 2: The Technical How Tos

So now you’ve decided that it’s time to use A/B testing in improving the quality and conversions of your site.  In the following installment, we’ll be taking a closer look at the tools you can use and provided a detailed “how-to” for the free and popular Google Website Optimizer. There are a number of commercial options on the market for A/B testing; in this article we’ll touch briefly on these options to help you make an educated decision. But The Google tool offers a full suite of options, is free, and integrates with other Google tools such as Analytics.

For these reasons, Website Optimizer is a great choice for most marketers or webmasters getting started with the A/B testing process.

What Is A/B Testing?

To briefly revisit, A/B testing is the process of putting forward two versions of something and seeing which one performs better. Common topics of A/B testing include headlines, placement of images, calls to actions, color schemes, and more.

Over time, marketers can use systematic A/B testing to find out which elements convert a higher percentage of visitors to their website, recipients of their emails, and viewers of advertisements. Systematic testing results in data-driven decisions about which campaigns should receive their focus and investment. This will help you attract more visitors, make more money, and have a more successful business.

Tools For A/B Testing

In this piece, we’re going to focus primarily on Google Website Optimizer which is a free tool that integrates smoothly with Google Webmaster tools and Google Analytics.

However, it should be noted that there are a number of commercial tools that are also available including Optimizely and SumoOptimize.  Technically, there are a hundred different software packages on the market that include different features and become increasingly complex technologically. We’ll explore a few of these options at the end of this article.

Getting Started With Google Website Optimizer


Before continuing with this article, watch this excellent, short introduction to the tool provided by Google.  It will contextualize the how-to and tutorial that follows here and give you a great head-start on understanding the flow of Website Optimizer which can be slightly counterintuitive until you get started.

To begin, visit www.google.com/websiteoptimizer

Sign in with your Google username and password.

Once you’re in, click on the Get Started button.  This will take you to the Google Analytics Terms of Service.  Read and accept those, and the click continue.

Once you click continue, this will take you to the Google Optimizer Dashboard. Select “Create Experiment” and then this will take you to the following page, asking which kind of experiment you would like to select.  Beginners should choose the A/B experiment.

Once you’ve selected the A/B experiment, this will take you to the A/B experiment check list which outlines some important data on information that you need know about how the A/B experiment operates.

The instructions will remind you that it’s important to choose the page that you would like to test, such as your homepage or a specific product detail page. It then advises you to go on and create alternate versions of your page to test. Finally, it recommends that you decide what page people will see once they convert, regardless of what page they come in from.

Once you reach this stage, stop.  If necessary, review my previous article about A/B testing .  Ensure that you’ve decided what variable to test and created two different versions of them, with different URLs.  Finally, make sure you’re clear on the conversion page that will come up afterwards.

Once you’re done, check the box next to “I’ve completed the steps above and I’m ready to start setting up my experiment” and click the “Create” button.

This will take you here:

Give the experiment a unique name such as “Sales Headline Test” and then enter the URLs of each of your test pages.  Google will verify that they are active.

Finally, enter your conversion page and click ”Continue”.

You will then be given the JavaScript tags necessary to be added to your website, or able to provide a set of instructions to a member of your team to handle the installation.

Once this is complete, you will return back to this same page to verify that installation is complete and to monitor the progress of your test.

Monitoring the success of your A/B test

Google recommends at least 100 conversions to be able to indicate a percentage of success.  In the simplest terms, you’re looking for which of the two pages encourages people to take the indicated action the most.  So if 100 people visit each of your pages, and option A converts at 4% while option B converts at 6% then option B is your “winner.”

But how many visitors are really enough to know when you’ve got a winner, versus a statistically anomaly?  The opinions vary, even among professional A/B testers.  A number of factors need to be taken into account to reach a valid sample size: these range from daily visits to the current conversion rate and the number of version you’re testing.

Two tools that can help you calculate the right information are the Google Optimizer Calculator and the A/B Experiment Duration Calculator. When possible, you want to aim for 95% statistical accuracy, with a 5% margin of error. It’s worth extending the test a bit longer before making your final decision.

Some hints to make your first test a success

  1. Test a page with high traffic level  When you’re ready to A/B test for real results, pick a page that receives a decent amount of traffic. The higher the traffic, the sooner you’ll reach a valid test sample and have a completed experiment. The results demonstrated from this experiment will help motivate you to incorporate testing into your overall strategy.
  2. Work with a high-conversion goal Along the same lines as choosing a page that gets a lot of traffic, it’s important to work on a conversion that people actually take. Adding items to a shopping cart, signing up for your newsletter, or viewing a critical section of your website are good goals. Test on actions that people take regularly, so it doesn’t take as long to get your data.
  3. Choose an important conversion goal Choose a conversion that matters, such as opt-ins or sales. Testing for the sake of understanding consumer behavior is interesting, but testing for the purposes of actually strengthening your business will keep you focused.
  4. Look at time on page as a conversion goal If your goal is to prevent bouncing or keep people reading your content longer, one option to consider is tracking time spent reading a page as a conversion goal. The full tutorial on how to do that is available here.
  5. Consider testing big items If you’re unsure of where to get started with your testing, a few obvious choices are headlines, product descriptions, graphics, placement of contact information, your call to action, and pricing. Each of these will yield meaningful information that you’re able to improve visitors’ experiences with right away.

Example tests

If you’re still struggling to conceptualize A/B testing or you simply want to see some tests in action, here are two great resources.

Which Test Won

Which Test Won is a site that shows you real life examples of A/B testing that was conducted by expert firms. You vote on which one you expected to have won.  You then get to see the real results as well as the case study behind the test. Spending some time on this site will help you build strong pattern recognition.

A/B Test

a-b test on writework a-b testing result

A/B Test is another site that allows you to look at case studies by type of test, view both versions, and then see which outperformed the competition. While it’s less interactive, the ability to get ideas by type of test can be very valuable to marketers interested in using testing to solve a specific problem.

Other A/B Testing Tools

Finally, I want to address the question of whether Google Website Optimizer is the right tool for everyone. The short answer is that I think it’s the right tool for the beginner and it’s the right tool for the average user. There are substantial advanced features that we haven’t touched on in this article and may explore for a future piece.

So in what instances do you need to consider other tools?

  1. If you’re going to be conducting ongoing, enterprise level multivariate testing for complex sites such as ecommerce sites with thousands of pages, a commercial tool might better fit your needs. (Consider Optimizely).
  2. If you’re opposed to corporate backed software, there is an open source tool called Genetrify.
  3. If you’re interested in more advanced testing of conversions along a funnel, Google Content Experiments (which Google is ultimately transitioning Google Website Optimizer) is the product to check out.
  4. If you are using WordPress and wonder how A/B testing can be done on WP, read Rochester’s article.

As you build your comfort with A/B testing, you can test additional variables and continue to iterate a page or website that dramatically increases your conversions, giving you more subscribers and better income.