6 Reasons Your A/B Test Failed (And What to Do About It)

11.11.20 By

Are your A/B tests ending with a shrug? Too often, A/B test results are inconclusive or worse, incorrect. A carefully designed A/B test avoids these common mistakes below.

1. You Didn’t Use Randomized Groups

An A/B test must be free of bias. By randomizing the placement of your test population into A and B groups, you can avoid the biases that are present in non-random selection methods.

You want your treatment groups to be representative of your entire population – and randomization is the best way to make that happen.

2. You Didn’t Include A Control Group

Knowing the factors you’re testing is important, but you can’t know how much of a difference those factors make if you don’t know the baseline. Before you start your test, take a snapshot of your current performance.

Then, make sure you have a control group – where everything stays like it currently is. That way, once your test is complete, you can get an accurate view of how your treatment(s) impact performance.

3. Your Variants Are Not Distinct Enough

For a test to be valuable, the factors need to be distinct enough to be worth testing. For example, if you’re looking for the best time to send an email, A/B testing 9 AM vs. 10 AM might not be as beneficial as testing 9 AM vs. 3 PM. While it is important to get very specific results if variables are too similar, your results might be biased or too subtle to be useable.

4. You Didn’t Have a Clear, Predefined, and Measurable Outcome

You can’t get value out of test results if you aren’t sure what you’re looking for. Using the social media example from before, it’s important to know what you want to see. Are you looking for a specific percentage increase in impressions? An increase in clicks?

While we always want to see improvement in performance, it’s important to know what that looks like. Just saying “improvement” isn’t enough. We have to know how much improvement, what metrics, if it’s repeatable, and more.

5. Your Hypothesis Wasn’t Clear (or you didn’t have one at all)

Similar to point number four, we can’t know how successful a test was if we don’t know what we’re testing for. Every test we do must be clearly and thoughtfully designed. Throwing things at the wall and seeing what sticks has its place, but when it comes to A/B testing, we need to be sure of what we’re doing.

Before you start an A/B test, make sure you have a clear, concise, provable, and actionable hypothesis. This will guide all decisions for your test, so it needs to provide a strong foundation.

6. You Over-Generalize the Results

Testing in a single market? That is a great way to minimize performance risk while gathering insights, but make sure that you understand the limits of your results. Just because an A/B test had a certain result in one situation/page/channel/region, it doesn’t mean that result is generalizable to your entire audience.

A/B testing is a powerful and effective tool when done correctly, and our analytics team is ready to help you prove your next hypothesis. Launch P.A.D.™ was built to help you set up and enjoy the benefits of an efficient, insights-driven analytics framework.

Learn more about LaunchP.A.D.™ and fill out the form to start making more informed data-based decisions with the help of our team.


We’re a team of digital customer experience (CX) gurus, passionate about helping businesses deliver experiences that delight through design, data, content & technology. We combine deep expertise with fresh thinking to deliver thought leadership that keeps you ahead of CX, DX & Automation (AI) trends.

We share actionable insights to elevate your CX and DX strategy, cutting-edge research and analysis, and inspiration to design experiences that win customer loyalty. Are you obsessed with exceptional experiences? We are here to help you deliver them. Connect with us.


Topics: Data & Analytics, Digital Marketing, Marketing Automation, Marketing Strategy

Solution-oriented technology is our specialty.