A/B Testing Guidance

Should this change be tested at all?

Not every change needs an experiment. Fixing a broken button or a misleading label does not need a test; it needs doing. Experiments earn their cost when the outcome is genuinely uncertain, when the change is expensive to roll back, or when the team disagrees and needs a fair way to decide.

Experiments also need enough traffic. A page with a few hundred visits a month may take far too long to reach a reliable answer. In that case, we recommend a small round of qualitative research instead, which can explain problems without needing large numbers.

What the test plan covers

Fixing the sample size before launch matters more than any other choice. Checking results repeatedly and stopping as soon as a difference looks significant inflates false positives, a problem explained clearly in Evan Miller's essay on the subject ¹. Our A/B testing guide covers that issue and the search engine rules in more depth.

Pre-launch setup check

Before the test goes live, we review the setup against the plan. We check that visitors are split as intended, that each variation renders correctly on common devices and browsers, that tracking fires once per event and that the primary metric can be calculated from the data being collected.

We also check the setup against Google's guidance for website testing, such as using temporary redirects for split-URL tests and pointing alternate URLs to the original with a canonical link. Those rules are summarized in our guide; the setup check confirms your test follows them.

Reviewing the result

When the test reaches its planned sample size, we review the outcome with you. The written review states the result on the primary metric, the guardrails, any segments the plan named in advance, and anything that may have disturbed the test, such as a campaign that sent unusual traffic. It ends with a clear recommendation: ship, discard, or retest with a revised hypothesis.

We are careful with segment results that were not planned in advance. Slicing results many ways will turn up apparent differences by chance. Those are worth noting as ideas for a future test, not as findings.

What we do not do

We do not sell or resell testing software and have no affiliation with any testing vendor. We do not build the variations; your team or agency does. And we do not promise that a test will win. A well-run test that shows no difference is still a useful answer, because it stops the team from spending more time on an idea that does not matter to visitors.

Timeline

We plan for one week to write a test plan and complete the setup check, and two to three days to review a finished test. The test itself runs for as long as the plan requires.