A/B testing

A/B testing is a user-experience research method used to compare different versions of a single variable. It is a type of randomized controlled experiment. In this setup, researchers typically use two variants, known as A and B. The goal is to determine which version is more effective at reaching a specific goal. This goal must be a defined, measurable outcome. Common outcomes include sales converted, registration rates, or clickthrough rates. A clickthrough rate is the percentage of people who click a link after receiving it.

The mechanism of an A/B test involves splitting a sample into groups. Each group is assigned a different version of a variable. For example, a company might test two different colors for a website button. One group sees a blue button, while the other sees a green button. To ensure the test is valid, the assignment must be random. This prevents bias from affecting the results. Researchers then monitor how each group responds to their specific version. By comparing the two, they can see which design performs better.

There are different ways to expand these tests beyond two simple versions. Multivariate testing, also called multinomial testing, allows researchers to test more than two versions at once. This can include using multiple controls. Another approach involves segmentation and targeting. In segmentation, researchers look at how different groups respond to the variants. For instance, a version might work better for men, while another works better for women. If a company finds these different patterns, they can use a segmented strategy. This means they send specific versions to specific groups to improve results.

History shows that the roots of this method are quite old. While it is hard to name the very first A/B test, randomized trials were used in 1835. In the early 20th century, Claude Hopkins used promotional coupons to test advertising. He described this in his 1923 book, Scientific Advertising. However, he did not use modern statistical concepts like the null hypothesis. In 1908, William Sealy Gosset developed the Student's t-test. This was a major step for statistical hypothesis testing. As the internet grew, A/B testing became a vital tool for digital companies. Google engineers ran their first test in 2000 to find the best number of search results. By 2011, Google was running more than 7,000 tests. In 2012, a Microsoft employee used testing on Bing to change advertising headlines. This single experiment increased revenue by 12% in just a few hours.

Statistical significance is a crucial part of analyzing A/B tests. It helps researchers decide if a result is real or just due to random chance. Different mathematical tests are used depending on the data being measured. For example, Z-tests are used under strict conditions regarding normality. Student's t-tests are used when conditions are more relaxed. Welch's t-test is a common choice for optimizing the mean of a metric. If a researcher is looking at binomial distributions, such as clickthrough rates, they might use Fisher's exact test. Other tests, like the Chi-squared test, are used for multinomial data.

A/B testing has many practical applications in the modern world. Social media platforms like Facebook, Instagram, and LinkedIn use it to study user engagement. They use it to see how new features affect user satisfaction. E-commerce websites use it to optimize the purchase funnel. Small changes to text, layouts, or images can lead to significant gains in sales. Even Google has used it to test dozens of different hyperlink colors. They wanted to find which hues would attract the most clicks to optimize revenue.

Despite its benefits, A/B testing has certain challenges and tradeoffs. One major downside is that tests are sensitive to variance. This means they require a large sample size to produce reliable results. While large social media sites find it easy to get many users, other companies may struggle. They might have to run the test for a longer period to get enough data. Microsoft developed a technique called CUPED to help with this. CUPED uses pre-experiment data to reduce variance, which requires fewer samples. Additionally, running tests can sometimes waste time and resources if the results are not helpful. In 2018, experts from 13 organizations, including Amazon and Netflix, identified several top challenges. These included issues with analysis, engineering, data quality, and deviations from traditional testing methods.