What is A/B Testing?

What is A/B testing?

In Digital Marketing, A/B testing (sometimes called A/B/n testing, split testing and multivariate testing) is the process of comparing variations of a piece of online content to see if any variation will perform better than the original for a given goal, with the targeted audience of the content.

Version A would be the current version, while version B through n each contain one or more elements that are modified from the original (for example, a different coloured call-to-action button), then the content visitors traffic is split amongst the versions of the content.

And since A/B testing is a simple controlled experiment, you can determine whether your content changes had a positive, negative, or no effect on the content visitors’ behaviour.

an A/B test of two variants of a product page (A and B). Variant A has a blue photo, while variant B has a red photo.

Example:

An A/B test of two variants of a product page (A and B). Variant A has a blue photo, while variant B has a red photo.

Testing more than one change of the content to find the ideal combination can also be done, it is then called multivariate testing.

A multivariate test tests variants of two or more elements simultaneously to see which combination creates the best outcome. Instead of showing which page variant is most effective (as in an A/B test), a multivariate test identifies the most effective variant of each element as well as analyzing the interactions between those elements.

Multivariate tests are useful for optimizing multiple aspects of a landing page, for example.

in a multivariate test you can test two headlines (original "H1" and variant "H2") and three hero images (original "A" and variants "B" and "C") simultaneously.

Example of a multivariate test:

You can test two headlines (original "H1" and variant "H2") and three hero images (original "A" and variants "B" and "C") simultaneously.

Another type of A/B testing is the redirect test, sometimes called a split URL test, allows you to test separate web pages against each other. In redirect tests, variants are identified by URL or path instead of an element(s) in the content. Redirect tests are useful when you want to test two very different landing pages, or a complete redesign of a page.

example of a redirect test: test a redesigned page hosted on a subdomain: Original – www.example.com Variant – new.example.com

Example of a redirect test:

Test a redesigned page hosted on a subdomain:

The original – www.example.com

The variant – new.example.com

In A/B testing, each variant is served at similar times so that its performance can be observed and measured independent of other external factors. You can also limit your test to a specific audience with audience targeting.

Although the A/B testing process is simple, the data that you can monitor is broad. You can use analytics tools for each variation test (for example: click tracking, conversion tracking, heatmaps, etc.) to help you find which variant performed better.

You can also use online tools to run your A/B tests. Many of the A/B testing tools are free like Google’s Optimize.

With the knowledge you gain from A/B testing, you can make incremental improvements to the performance of your online content, and you can anticipate which changes will be more noticeable (for better or for worse) to your target audience.

improvements can sometimes be seen through testing elements like copy text, layouts, images and colours. In these tests, users only see one of the two versions, as the goal is to discover which of the two versions performs better.

A/B testing is a great way to quickly find the result of the comparisons you make so you can decide if you need to do more tests to validate it, or if you should move on to the next variable test.

Contact us now to get help with your A/B testing!

Why is A/B testing important?

In Digital Marketing, data is what drives performance. In other words: you need to measure the performance of your online content and always try to improve it. A/B testing is one of the best methods to improve the performance of your online content as measured by your KPIs.

There are many reasons why to do A/B testing, for example: to create a better user experience, to increase your content’s conversions or leads, and to decrease the numbers of visitors leaving your content early (also known as the bounce rate).

Here is an example of the importance of A/B testing and its impact on the profits of the Microsoft company: In 2012, a Microsoft employee working on the search engine Bing created an experiment to test different ways of displaying advertising headlines. Within hours, the alternative format produced a revenue increase of 12% with no impact on user-experience metrics. (Source: Kohavi, Ron, and Stefan Thomke. 2017. “The Surprising Power of Online Experiments.” Harvard Business Review 2017 (September): 74-82)

A/B testing can be performed continuously on almost anything, especially since many marketing automation software now comes with the ability to run A/B tests on demand.

We can help you get started with your A/B testing today, just contact us now!

What types of content can be A/B tested?

Almost any piece of online content can be A/B tested. You can test written content (like articles), visual content (like pictures and video) and even audio content.

You should first focus on the most important elements of your content and test them first. Start by testing things like headlines and copywriting, calls to action, images, audio and video, forms, email subject lines, content length, product descriptions, social proof (user testimonials and ratings) and marketing emails.

Some other common elements that you can A/B test include layout choices (placement of page elements) and functionality choices (interactive elements such as error messages).

You can also, for example, test the performance of the navigation of your website by changing a link’s text or a button’s colour.

You can also test the performance of things like the loading speed of your page by changing the size of a picture for example. You can also test marketing emails and ads.

A/B testing can also be used by product and applications developers and designers to demonstrate the impact of new features or changes to a user experience.

How to perform an A/B test?

In online content A/B testing, content visitors traffic is split amongst two pieces of online content: the original content A and it’s content variant B. Then you compare their performance using a few selected KPIs to know is the B variant performed better than the original.

Testing is done by showing the original content and it’s changed version to the divided visitors of your content at the same time and then monitoring the performance of the variant to find if it improves your selected key performance metrics (KPIs). You can keep track of your selected KPIs by using a tool like Google Analytics (which used to have an A/B test tool).

For example, when you have published an article on the web, you can use A/B testing to find out if using a different title will get more visitors to stay and read the article. Or you can test if changing a certain word in your video title would get more viewers to click and watch it.

It is recommended to wait for your content variants to receive at least 1,000 viewers and to run your test for at least 7 days to account for day to day fluctuations, you can increase the test duration in 7-days increments.

And as good practice, try to limit the changes to the B variant to the minimum necessary for the test, even when you are testing multiple variables. So, if you are testing the title, for example, change one or two words only whenever possible.

We can help you create your A/B testing plan, contact us now!

Example A/b Tests

What you decide to test depends upon your goals and priorities. While not a substitute for a test plan, the following example experiments can help get your testing started.

Pages to optimize

Before performing tests on your most popular web pages (like your home page) consider starting small first, which will allow you to experiment with lower-risk pages. Keep in mind that you’ll need to test pages with enough traffic to get results quickly.

Start by experimenting with the colour of a button, the copy in a call to action, or the order of a navigation bar. After completing a few experiments, graduate to testing headlines, images and landing pages when you’re ready.

Here are some web pages that are prime candidates for optimization and experimentation:

  • Category/family page

  • Product page

  • Cart page

  • Thank you page

  • Contact form

  • Search results pages

  • Landing page

  • Home page

Elements to optimize

The following are some examples of website elements that are frequently tested, but this list is by no means exhaustive. Use it as an idea-starter to brainstorm on potential experiments for your website.

Navigation

  • Copy – Voice, tone, message.

  • Appearance – See colour and size examples.

  • Order – Link order can make a big difference.

  • Design – Color, depth, size.

  • Sub-navigation vs. not.

  • Search

  • Features – Suggestions, type-ahead, auto-complete.

  • Box location

Design

  • Size

  • Advanced search vs. not.

  • Results on the same page vs. new page.

  • Button type – Words vs. icons.

Forms

  • Quantity of form fields – Less is more.

  • Required fields – or not.

  • One page vs. multi-page.

  • Copy – Headline, label, buttons, help text, tooltips.

  • Design – Simple vs. complex.

  • Step indicators – Numbers, breadcrumbs, percentage complete.

Buttons

  • Button type – Words vs. icons.

  • Size – Can you see it from a couple of steps back?

  • Color – Green and orange historically test well, but it depends on your colour palette.

  • Copy – Less is more. Use the active voice.

  • Placement – Is it above the fold?

  • Images – Do they convert?

  • 3D vs. flat – Timeless vs. trendy designs.

Dialogues

  • Overlays

  • Modals

  • Roadblocks

  • Accordions

  • Size

  • Timing

Content

See Colors, Buttons, and Images sections above for more ideas.

Videos

  • Cost-benefit analysis of including videos or not.

  • View portal size.

  • Location on the page.

  • Thumbnail image.

  • Title and description.

  • Autoplay vs. click to play.

  • Auto sound vs. click to hear.

  • Sound volume settings.

Trust elements

  • Quantity – Single versus multiple.

  • Security icons – Thawte, McAfee, etc.

  • Privacy text – Policy links and email disclosure copy under email capture fields.

  • Testimonials – or not.

  • Social proof – The positive influence created when someone finds out that others are doing something.

  • See Buttons section above for more ideas.

General areas to optimize

Some general areas that can be tested and optimized on every website include the following:

Copy

  • Wording – Put yourself in your reader's shoes, try a different approach.

  • Clarity – Simplify complex sentences, remove jargon.

  • Length – Write shorter instead of longer.

  • Format – Use action verbs ("Try it now.")

  • Voice – Is it on-brand, consistent, use active vs. passive voice.

  • Fine print/footer links – Make them descriptive, don't obfuscate!

Images

  • Quantity – Try less if you have a lot

  • Sizes – Smaller or larger?

  • Subject matter – People, babies and pets.

  • Products vs. features – Both a popular in images

  • Carousels vs. static images – The result may surprise you.

Colours

  • Trendy vs. classic.

  • On-brand vs. off-brand.

  • Call To Action (CTA) – matching your palette vs. standing out.

  • Textured/shaded vs. flat.

  • Visually-impaired/ADA compatibility.

Contact us today to help you with your Digital Marketing needs.

About the author:

Seiffeddeen Keilani is a Google Ads specialist and founder of the S2HA2R (www.s2ha2r.com).