When You Should Give A/B Testing a Try
What, Why, How is A/B Testing?!? Unsure about this topic? We're gonna cover almost everything that you need to know!
Have you ever had a situation where you have 2 headlines and you don't know which one to choose? Or, how about you have 2 pricing but don't know which one people would want?
This calls for A/B testing and it does just that! It helps you test out your different ideas and see which performs the best!
Here are the key topics we'll talk about:
A/B testing is a marketing experiment to compare and test several variations, e.g. a headline, visual, copywriting, to determine which performs better.
It is done by splitting the audience equally to reduce bias and success is measured by the number of interactions and conversions.
We often have many ideas, but we are not sure what our audiences truly resonate with; which is why, we use A/B testing as a marketing experiment to determine which idea or variation works best.
A/B testing allows you to experiment on copy, video, visuals, testimonials, headlines, layouts, user experience (UX) and other elements. Sometimes a simple change can create a huge impact on conversion rates which would ultimately affect the bottomline.
Here is what a friend from AirAsia said (AirAsia is Asia's biggest budget airlines):
We A/B testing a lot as even small variation could impact our bottom line (profit) upwards in the scale of millions.
It could be as simple as changing the colour of the button or the text on Call-To-Action. To remove the guess works out of which color, text or visual performs better; A/B testing is used for this purpose.
Basically, you cannot run an experiment of 20 people and assume that the results will be conclusive for 1000 people.
With Facebook advertising, they suggest 50 conversions over a period of 7-days for them to move past the learning phase and choose the winning ad to focus on.
Here are some scenarios/ examples on how A/B Testing can be conducted.
Imagine you have an email list of 10,000 people and you want to send an email out. You have 2 headlines and you are not sure which one would have a better open rate.
A common practice for email would be:
Conversion in this experiment: Open rate
Outcome: The winning headline will be sent to the remaining 9000 people
Imagine you have an email list of 10,000 people. You have 2 email copies and you are not sure which one would have a better CTR.
A common practice for email would be:
Conversion in this experiment: Click-through-rate
Outcome: The winning content will be sent to the remaining 9000 people
Similarly to the above examples, you can create multiple versions of the elements you want to test.
Common conversion goals:
Outcome: The winning element will become a permanent fixture on the page
Facebook is similar as the above and it even has the A/B testing feature for you
Common conversion goals:
Doing an A/B test requires you to have proper tools.
Type of test: A/B Testing, Multivariate testing, split testing
There are several types of test which you may have heard of:
While A/B testing is great fun, here are some things to look out for to ensure your marketing experiment is a success!
If you do not have meaningful traffic, you would not be able to even run an experiment that would yield a conclusive result.
You would want your test to be statistically significant. Use this tool to check: https://neilpatel.com/ab-testing-calculator/
If you don't know what goals you are trying to achieve, it would be pointless to run an A/B test. You should also make sure the conversion goal makes sense.
For example: If you are testing email headlines, the goal should be open rate, not email click-through-rate.
If you are not planning to take any action at all, save your time and do something else instead. :)
Now that you know the concept behind A/B testing, it is time for the science behind it!
According to towardsdatascience.com,A/B Testing is similar to hypothesis testing.
Here are a few things to keep in mind:
Notice that it should follow this pattern:
There are several randomization and cluster strategies available, but in this case, many of the tools such as Google Optimize or Email softwares do a random sampling strategy.
Within the week, there are 10,000 visitors to your website and the software will randomly distribute the people.
Why should we randomise?
You can't be taking a sample size of 10 people and assume it to be true for 10,000 people right?
You may have heard of p-value.
P-value means "In statistical hypothesis testing, the p-value or probability value is the probability of obtaining test results at least as extreme as the results actually observed during the test, assuming that the null hypothesis is correct."
In other words, how statistically significant is the result. The more significant, the higher the probability that the result holds actionable recommendations.
Look at the 2 examples below. Both have the same conversion rates.
Often, you do not have control over the number of people coming into the website or what not.
How does softwares such as Facebook determine significance?
Tools such as Facebook Ads would have a learning period where they would want to obtain at least 50 conversions within 7-days to be able to safely conclude which ad is truly working before showcasing the result to you OR optimising the ads for you.
Measure the wrong thing and your test would not make sense.
Let's just say that my landing pages are poised to collect leads, but what happens if I ended up using "purchase" as my measurement for success?
Understanding the test design is important, here is an example:
This would mean that by me using Purchase as a measurement of success would not be reflective on the test because there are added variables that have influenced the outcome.
A/B testing is a powerful methodology and tools have made it simple for us to execute and take actionable moves for marketing purposes.
Nonetheless, understanding the statistics and data science behind A/B testing would open up ideas on how you can better test your market.
Just one thing though, these A/B testing activities require you to have good amount of traffic to make it work.