In its simplest form, A/B testing (also known as split testing) is the process of comparing two variations of a marketing asset, e.g., a webpage element, call-to-action, landing page, email, etc., usually by testing users’ responses to Variant A vs. Variant B and concluding which of the two variants is more effective (see figure above).

Why You Should A/B Test Your Marketing Activities

Many marketing departments rely on A/B testing because it is one of the most effective ways to tailor your approach to customers. While data and trend analyses are important elements in designing engaging web pages, A/B testing provides practical, tangible evidence of your most impactful marketing techniques.

Modern online metrics and analytics also make it easy for you to evaluate the results of your A/B testing. With analytics, you can easily monitor key metrics, including:

• how many people were exposed to each version,
• how long they interacted and engaged and most importantly,
• what percentage completed the campaign goal and converted.

An A/B test takes a significant amount of guesswork out of marketing. For example, if you think of a new marketing approach that could improve your company’s performance, you can set up an A/B test and then monitor the data. If you’re right and the new option gets better results, you can move forward with your new plan. If the original technique still performs better, you can continue with your regular routine and think of different marketing approaches in the future. Ultimately, A/B testing helps you make decisions based on data. This way you know when a particular marketing tactic or campaign is working. Or, more importantly, when one is failing.

Since every organization is different, don’t assume that what worked for others will automatically work for you. When you A/B test your marketing activities and content, experiment with different ideas. This is useful because sometimes a small change can make a big difference in the results you get.

Common Areas to A/B Test

While not exhaustive, the following list presents several common key areas where marketers A/B test:

• Headlines and subheadings
  Whether it is the “titles” on a webpage, headlines for a piece of content, or the subject line of your email, you may want to experiment with these to see if small changes make a difference in engagement rates.
• Copy
  Copy is simply the text that you have written in your marketing materials. While copy can include the words and text itself, copy can also include the location of your text. Presenting two options and seeing which version is more engaging and/or converts more people could be an area to explore.
• Form design
  When you look at your forms and the fields to be filled out, think about the number of fields and the information requested to see what potential leads submit and when completion rates drop significantly.
• Calls to action (CTA)
  As marketers, having compelling calls-to-action (CTAs) is critical to getting leads to convert. Testing a variety of options provides data on which words deliver the best results. Here’s an article explaining how you can test CTAs using Hubspot.
• Images
  Like CTAs, you can test different images and image types. For example, try different photos to see which one(s) capture your audience’s interest more. You might even explore different types of imagery, e.g., photos, icons, clip art, etc.
• Colours
  And, finally, sometimes colour can have an impact in how your target audience behaves. Testing contrasting colours, brighter colours, etc. can often provide insights as to what catches your target audience’s attention and what they prefer.

For more A/B testing ideas, here’s an article, 60 A/B Testing Examples to Get You More Conversions.

How to Conduct an A/B Test

Now that you know what an A/B test is and several potential areas to test, let’s discuss how to conduct an A/B test. If you’re ready to begin taking advantage of A/B testing to find the best marketing activities for your organization, here is what we recommend:

1. Gather insights
   Before starting your split tests, gather any information you have about your customers. This can include both direct observations about your customers and industry insight data about ideal characteristics related to your target audience’s preferences. The more detailed your understanding of potential customers, the easier it will be to get positive results from your campaigns. Remember your customer personas?!
2. Set your goals
   Not all marketing activities have the same goals. For example, the metrics you will monitor for a brand awareness campaign will be different than those for a campaign focused on maximizing sales. Having a clear goal in mind makes it easier to know which metrics to focus on. We will provide more examples and details in the Digital Marketing Analytics chapter.
3. Create a theory
   Once you have your target audience defined and a goal in mind, you can create a marketing theory. This is often called your “hypothesis”. Determine what you think will be an effective method for your target audience to achieve the goal you are seeking. This theory will set the framework for your different testing options. For example, your hypothesis might be that the colour of the CTA button affects the number of clicks and conversions.
   
4. Build your variations
   Although most content systems will allow you to perform A/B tests with many different variants, it is usually best to limit each round of testing to two or three variations. This allows you to quickly hit sufficient traffic numbers or volume for each option so you can compare them. It also makes it easier to isolate differences in the options to determine which technique led to better results.
5. Run A/B tests
   After building your variants, it is time to launch them and allow the A/B tests to run. There is no perfect sample size. Instead, the right size depends on your marketing priorities. Targeting a smaller sample size will allow you to analyze your results and adjust more quickly and at a lower cost, while targeting a larger sample size provides you with more information to support larger-scale decisions. Clearly, there are pros and cons to each approach. So, the decision will be based largely on your goals, objectives, and levels of confidence required.
6. Analyze the results
   Once you have gathered enough data to make an informed decision, you can begin looking at the data gathered. Make note of the key metrics you are tracking as a priority, but do not entirely ignore other metrics. For example, if you are running a campaign to build brand awareness and discover that one version is generating significantly higher sales numbers, while this may not make it the ideal option for the current campaign, it still provides valuable information you can use to monetize other campaigns. The winning option is the one that is performing best, and that version should become the basis for your activities as you move forward.
7. Make adjustments as needed
   After you discover which option won the A/B test, adjust your activities as needed. If the metrics for the winning variant are clear, expose your entire target audience to that option. If it is somewhat unclear which option is clearly the winner, perform a new A/B test by adding in new options and test against the leading variation. When testing new versions against the previously leading variation, take results from the first test into account when deciding on the best option.
8. Continue monitoring
   Even after multiple rounds of A/B testing, you will still need to regularly monitor your marketing activities to ensure they are still effective. Marketing tactics will commonly reach points of diminishing returns, either as the content becomes less relevant or you begin to run out of potential ideas for a specific campaign. Make a habit of monitoring the campaign’s performance even after you have found a winning variant and exposed the entire audience to it. Should performance begin to slip, you can end the campaign or return to Step 1 and go through a new round of A/B testing.

 For a quick summary of A/B testing, here’s a 4-minute video, What is A/B Testing? | Data Science in Minutes:

One additional term you should be aware of is multivariate testing. Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations (see image below).

Tools for A/B Testing

Now that you understand what A/B testing is and how to conduct one, the following links provide a few frequently used tools for A/B testing:
(Please note that the tools listed below are all free or offer free trials.)

• Google Optimize – Google Optimize is a Google tool that lets you test and tailor different variations of your website. The main selling point of Google Optimize is its native integration with Google Analytics, allowing you to gain automatic access to rich behavioural insights. You will also be able to target the valuable segments you have already discovered using Analytics. Try out Google Optimize for free and upgrade to Google Optimize 360 for enterprise-level website testing and personalization. Here’s a quick video overview:
  
• Kameleoon – Kameleoon is an advanced optimization platform that offers a variety of features. Using Kameleoon, can run advanced A/B, split, and multivariate tests quickly and easily. You can also see key user insights with their navigation analysis tool and test different user segments with over 40 targeting criteria. Dynamic traffic allocation algorithms also make sure your traffic is optimally divided in order to shorten your decision cycle and improve return on investment. You can get started on Kameleoon by creating a free account and upgrade to an enterprise plan if the tool suits your needs.
  

   

• Symplify (formerly SiteGainer) – Symplify is a company that offers a full suite of conversion optimization tools, including A/B testing, multivariate testing, personalization, heat maps, popups, and surveys. They also have a team of experts to assist you with test ideas, design, programming, analytics, and personalization. Symplify offers a free trial. Pricing plans are based on monthly traffic, payment schedule, and additional services.

For more options, reviews, and pricing details, please check out this article, 24 A/B Testing Tools. In this article, they make the following recommendations:

• The Best Solutions for Large Businesses:
  1. Symplify
  2. Optimizely
  3. Google Optimize 360 (the integrations alone make this a great option)
• The Best A/B Testing Tools for Marketing Teams:
  1. VWO (if you have the budget)
  2. Convertize
  3. Freshmarketer
• The Best A/B Testing Tools for Small and Medium-Sized Business:
  1. Convertize
  2. Zoho Pagesense
  3. OmniConvert
• The Best Landing-Page Tools:
  1. Unbounce
  2. Clickthroo

The A/B testing software market is in a healthy place, with solutions for every size and type of business. Most of the options in the above list^[1] include a free trial (some as long as 60 days), so there is no reason not to try them for yourself.

Media Attributions

• A/B testing overview by guest author on Techno FAQ is licensed under a CC BY-NC-SA 4.0 licence.
• What is A/B Testing? | Data Science in Minutes by Data Science Dojo is licensed under a Standard YouTube License.
• Multivariate A/B tests by Navot is licensed under a CC BY-SA 4.0 licence.
• Google Optimize Tutorial by Loves Data is licensed under a Standard YouTube License.
• What is personalization? by Kameleoon is licensed under a Standard YouTube License.