AB Testing, sometimes called Test and Learn, is the a method of showing different content or layouts to your website visitors.
AB testing is one of the best ways you can understand user behaviour on your website. It allows you to run different tests, to evaluate your hypothesis on your visitors behaviour.
As your website traffic grows, you will be able to run a number of AB tests, or run tests which you can get results on very quickly.
Key takeaways- AB Testing
Top 5 tips for AB Testing
See our checklist
1. Have a good reason/hypothesis for your test
2. Put goals and measures in place to understand your test’s success
3. Know how long to run the test for, and ensure it is deployed quickly without weeks of development
4. Make sure your test is big enough so people notice and can be tested quickly BUT not too big it will lose you money
5. Ensure you only change one thing before moving on, otherwise you won’t know what worked!
AB Testing
The What, Why and How
What is it AB Testing?
It is a way of allowing you to build tests to validate your to see whether those things have made a difference to your business.
It allows you to shortcut your learning into why your visitors are doing certain things.
It is a rapid, not a highly structure development process, focused on optimising your website, using data to validate what worked and didn’t work.
There a number of tools which help you deliver this. Some of the tools we have used include: Google Optimize (free option with integration with Google Analytics), Monetate (built for marketers), Attraqt (for testing merchandising).
For a larger list go here.
Why should I measure AB Testing?
It will allow you to validate new designs, before they are rolled out and the potential impact they might have whether that be positive or negative.
It would allow you to test different user journeys throughout your website
It reduces your risk by testing on a small audience, knowing whether if it has an impact prior to a rolling it out across the whole of your customer base/website visitors.
How to get started with AB Testing?
To carry out a successful AB Testing programme you will need to be prepared to create your plan first.
There are 8 steps to creating a successful test and learn program:
- Define your test hypothesis and channel (email, website, ads, social)
- Set out your measurement success criteria for your hypothesis
- Understand the volume of traffic and how long the test should run for
- Set out your priorities and the order in which your test will be run
- Mock up design prototypes (if required) ready for building
- Create the tests in your development environment and test they work
- Roll out your tests in a live environment and evaluate their performance against your normal traffic
- If the tests are successful switch over the learning/test into your live or production website.
Want to learn more?
See our comprehensive video guide to creating your AB testing plan
On the go?
Learn about AB testing in our 15 minute audio walkthrough.
Don't forget to check out our book - Your Number's Up!
Getting a grip on Data and Measurement to accelerate your Direct to Consumer ecommerce sales
With over 280 pages our book is full of actionable material and Avinash Kaushik has kindly written our foreword.
We have created a series of tools to help you understand, measure and grow your ecommerce business.
These include:
-
Defining your digital measurement maturity
-
4 x Cheat sheets for improving your ecommerce measurement
-
A full glossary of over 60 metrics, understand what they mean and why they are important
-
Practical scenarios for diagnosing your ecommerce website
-
Common campaign measurement mistakes and challenges and how to overcome them.
Before we start, when should you do AB testing?
1. You have a hypothesis you need to validate and test
2. You can create a test (with a testing tool) which requires no core code changes to your website platform
3. The creation of your Test and resources should be less than 2 weeks
If the above are true you are ready to get your first AB test underway.
Our 8 step plan to creating a successful AB testing program
Step 1 of 8 - Define your test hypothesis
The first step involves outlining what your test needs to prove.
Your tests could come from any part of the organisation. They might include improving your marketing effectiveness, showing the right products, or seeing which content best engages your audience. Remember to categorise your hypothesis so you know if it is testing User Experience (UX), Website functionality or Content vs Product.
Make sure you capture all the relevant information about your hypothesis. This will allow you to set the right metrics (in step 2) and also help your prioritise this hypothesis (in step 3).
Step 2 of 8 - Set out your success criteria
Now you have defined your test and the criteria you want to prove, you now need to categorise your metrics.
If you set the metrics correctly know you will firstly know whether the test has been a success AND what behaviours it has also influenced.
Start with setting your primary metric. The one thing you want to prove. In this example we have looked at Product Viewed Rate, Add to cart rate and Cart completion rate.
Once you have your primary metric set your secondary metrics which will allow you to understand why things happened. These might include page load time, bounce rate or returning visitors as some example. For some help on defining these go here.
Step 3 of 8 - Understand the volume of traffic and test duration
Test length is determined on the amount of traffic your website gets coupled with the items described here, Page, Device and Number of Tests.
To ensure you have a reliable test ensure you have enough traffic to test, but do not deploy it to the whole website as this could have a major impact.
There are a number of testing calculators out there to help you define if you test is reliable (in data terms call significant).
1
Define the pages which the test will run on. Lower level pages (those accessed from menus) will have less traffic and require the test to run for longer.
2
The device which the test will run on will impact the test length. If you run it on all devices you will see a shorter time to results. Choosing a low volume device (such as Tablet) will take a lot longer.
3
Testing different things is a great idea, but the more tests the more data you will need. Therefore try to only create multiple tests on large volume pages.
Step 4 of 8 - Set out your priorities
We use the following criteria to evaluate each tests.
The criteria can be changed, but it is important to balance the business impact against customer benefit. And then look at cost and speed to market.
We then upweight the organisational focus for each of the criteria to ensure each test is focused on the right outcome. When you combine both you get an overall priority.
For a more detailed walkthrough of creating your priorities go here.
Step 5 of 8 - Mock up design prototypes
This is an optional step if the changes you are making are not visible to the customer.
These could be a change of website functionality such as changing the default sort order or a product listing page.
The images here show the different Proposed designs. Each audience (visitors) which see the test a mutually exclusive i.e. they only see their test design.
1
This is the A test. Or the current page, which you are testing against. In this example the test was to see if changing the “burger Menu” in the top left could get more clicks, as visitors were lost.
2
Option 1 “Test B” was to create a menu with the word Menu on there.
3
Option 2 “Test C” was to create a menu with the word Menu on there, but with a different colourway.
Step 6 of 8 - Create the tests in development
Before deploying the test in the live environment set up your test in a development environment where only the internal team can see the test in action and the data from the test can be captured.
Step 7 of 8 - Roll out and evaluate your live tests
Once you have launched your tests, keep an eye on the performance of the test. This should be compared with the generic/current experience.
Review the category AND the outcome of each test. Compare the results of the specific test and if this improved or declined the outcome.
NOTE: Quite often the goal/KPI of the test might not improve but other metrics might be effected.
Step 8 of 8 - Push your Test into live
Now the test has been proven to be successful, schedule the test to be included into your live/production environment/website.
Follow your normal website release process and ensure any shortcuts are handled and upgraded prior to releasing the change.
People also ask . . .
Why bother with AB Testing
AB testing has a number of benefits. The reasons you should bother with it include:
- Limited exposure: You will be able to test your hypothesis in a safe environment. If you get it wrong then you have only impacted those who have seen the test rather than the whole audience or customer base.
- Huge payoffs: While an individual test is unlikely to have a big impact on it’s own, the impact of many successful tests can be massive. Imagine increasing your conversion rate by 0.5%
- Lower costs: By focusing on very specific hypothesis and using tools which do not require large amount of development resources means you can test your website functionality and look and feel, without big development overhead and costs.
- Faster deployments: AB testing by it’s nature is all about rapid learning. As the marketing team will (mostly) be able to create their own tests, the results should be able to found very quickly. Not only with a reducing development time but a quick understanding of the test’s impact.
- Data driven evidence: AB testing provides large amounts of data. As long as the test has been set up correctly and the right information has been captured the business case should be straightforward, allowing the marketing team to influence the larger website development roadmap/release cycle.
How long should I run by AB test for?
There is no single number of this.
What you will need to do is understand how much traffic you need to provide you with “confidence” the test is representative of your audience/website visitors.
Use this tool to tell you how many you need
How do I track my AB Test
Ensure you have a tool in place to be able to run AB tests.
Once you have deployed your test make sure you are able to see them metrics for each test, without this tracking you will not know if the test has made a difference.
The tracking should be available inside your AB testing tool.
If this is not available make sure you track the tests inside your web analytics tool. This could be as simple as creating two different landing pages with different names and seeing the conversion rate, add to cart rate, average order value of these pages compared to the generic or current page.
When should you not do AB Testing?
Do not run AB testing if:
- You do not have a solid hypothesis backed up with data
- There is no AB testing tool in place to help you randomly split your traffic
- You do not have metrics or tracking in place to know if the test worked
- You development process is slow and tests take longer than 2 weeks to run
- You have a small amount of traffic, as tests will need a long time to be proven to be accurate or reliable
What does your AB Testing tool need to do?
Your tool will need to take the labour out of the data collection and analysis.
The type of things which we find makes a useful tool includes:
- The ability for the tool to to automatically split your traffic for your AB test
- Notifications if your AB test has reach significance/reliable results
- Alerting you if there is a major drop in conversion, average order value or increase in page load time or bounce rate.
- Present the results of your AB test with the change in the primary objective and secondary metrics which relate to your test
Recommended Reading
Below is some recommended content to help you with your analysis in marketing. This content is mixture of content, tools or our book to help you get more out of your digital marketing and data in your business.
Read our book Your Number’s up! – Getting a grip on Data and Measurement to accelerate your Direct to Consumer ecommerce sales
Learn how you can measure and grow your ecommerce business. Theory and practice all in one book.
Define: Conversion Rate – Read about a critical metric for measuring the number of people who take action on your website. Learn how to diagnose conversion rate and understand how conversion rate impacts your cost per acquisition.
Discover: Resource for senior marketers – See our latest research or white papers to help you get ahead. Alternatively, read our digital marketing glossary for senior leaders to understand the jargon and acronyms of digital.
Index