How to Run A/B Tests Without Hurting Search or Your Statistics

Two ways a test goes wrong

Search engines index your pages and rank them for searches. A test that changes what pages show, or redirects visitors to alternate URLs, can look to a search engine like an attempt to show it one thing and visitors another. Google publishes guidance on how to test without that risk.

Statistics decide whether a difference between variations is real. The most common mistake is checking the results repeatedly and stopping as soon as one variation looks like a winner. That habit can turn random noise into apparent wins at a far higher rate than the testing tool reports.

Google's rules for website testing

Google Search Central's page "Minimize A/B testing impact in Google Search", last updated December 10, 2025, sets out what Google expects ¹. The key points, quoted from that page:

Do not cloak

Google's page warns against showing "one set of URLs to Googlebot" and a different set to people. Cloaking means deciding what to show based on whether the visitor is a crawler. Testing tools that exclude crawlers from tests, or always serve them the original, risk being treated this way. Let crawlers fall into variations the same way people do.

Use canonical links on alternate URLs

If a test uses separate URLs for variations, Google's guidance is: "We recommend using rel="canonical" rather than a noindex meta tag". Each alternate URL should point to the original page with a canonical link, so search engines understand which version to index.

Use temporary redirects

For tests that redirect some visitors from the original URL to a variation, Google advises to "use a 302 (temporary) redirect, not a 301 (permanent) redirect". A 301 tells search engines the move is permanent. The same page notes that JavaScript-based redirects are also acceptable.

End tests promptly

Google asks site owners to "remove all elements of the test as soon as possible" once a test concludes. It also warns that if a test runs for an unnecessarily long time, "we may interpret this as an attempt to deceive search engines".

The peeking problem

Most testing tools show a live significance figure. It is tempting to watch it daily and stop when it crosses 95 percent. Evan Miller's essay "How Not To Run an A/B Test", published in 2010, explains why that ruins the result ².

A significance test assumes you decided the sample size in advance and looked at the result once. If you look repeatedly and stop the first time the figure crosses the threshold, you give random fluctuations many chances to look like a win. Miller shows that the reported significance becomes meaningless under those conditions. In one of his examples, repeatedly checking a test that had no real difference, with a 5 percent significance threshold, produced a false positive rate of 26.1 percent, more than five times the rate the threshold promises.

Miller's fix is simple: decide the sample size before the test starts and do not act on the result until that sample is reached ². If you need to monitor tests continuously, use a method designed for it, such as a sequential testing approach, and use it as designed.

Writing a test plan

A written plan protects against both search and statistics problems, because it forces the key decisions to happen before anyone sees data.

1. Hypothesis. The change, who it affects and why you expect it to help.
2. Primary metric. One measure that decides the result.
3. Guardrail metrics. Measures that must not get worse, such as refunds or page speed.
4. Sample size. Calculated from your current rate, the smallest difference worth detecting and the significance and power you want.
5. Duration. At least one full weekly cycle, often two, so weekday and weekend behavior are both included.
6. Search setup. Which of Google's rules apply, and how the setup follows them.
7. Stop rule and decision rule. When the test ends, and what result leads to which decision.

Our A/B testing guidance service writes and reviews these plans.

Speed and layout shift during tests

Client-side testing tools often hide the page briefly while they swap in a variation, or change content after the page has loaded. Both can harm the experience. A hidden page delays loading, and content that moves after loading adds to Cumulative Layout Shift, one of the Core Web Vitals. Measure page speed and layout stability for each variation as guardrails. Our Core Web Vitals guide explains the thresholds.

When not to run a test

Tests need traffic. If a page has too few visits to reach the planned sample in a reasonable time, the test will either run for months, which Google warns against, or end early, which Miller warns against. In that situation, qualitative research is a better tool. A handful of usability sessions can explain why a page underperforms without needing statistical power, as described in our sample size guide.

Some changes do not need a test at all. Fixing a broken link, a misleading label or an inaccessible form should simply be done.