Why your site probably cannot be tested

Split testing is presented as the rigorous way to improve a site, and on a small site it is usually a way of generating confident conclusions from noise. The arithmetic is unforgiving and worth facing before spending a year on it.

Why most small sites cannot test

A test has to distinguish a real difference from chance, and the smaller the difference the more observations that takes. Detecting a modest improvement in a conversion rate of a few per cent takes thousands of visitors per variation, which a site with a few hundred visitors a week does not have in any useful timeframe.

What happens instead is that a test runs for two weeks, one version is ahead, the test is stopped, and the difference was chance. Then the change is kept, the next test is run against it, and after a year the site has been rebuilt on a series of coin flips.

What to do instead

1. 01Fix the things that are definitely wrong. A cost revealed at checkout, a form with nine fields, a page that does not state the price. These do not need testing; they need fixing, and the effect is larger than anything a test would find.
2. 02Make large changes rather than small ones. A test can only detect a difference it is powered to see, so a rewritten page can show an effect where a button colour never will.
3. 03Watch the step-by-step drop instead. A funnel tells you where people leave without requiring a comparison group at all.
4. 04Ask five people to use the page while talking. Qualitative, unscientific, and it reliably finds the two problems that a year of testing would eventually confirm.
5. 05Compare before and after over a long period, accepting that other things changed. Weaker evidence than a test and stronger than a badly powered one.

If you do test

• Decide the sample size and the duration before starting, and do not look until then. Stopping when you like the result is the most common error and it guarantees false positives.
• Run for whole weeks. Behaviour differs by day, so a test over ten days is comparing different mixtures.
• Test one thing, and make it something substantial enough to matter.
• Write down in advance what you will do in each outcome, including the outcome where there is no difference, which is the most likely one.
• Expect most tests to find nothing. That is the normal result and it is not a failure of the test.

The honest summary

Testing is a tool for sites with enough traffic to resolve small differences. Below that, the way to improve a page is to find what is broken, fix it, and watch the funnel. That is less satisfying than a dashboard reporting statistical significance, and on a site with two hundred visitors a week it is the only honest option.

Watching five people, properly

The replacement for testing at a small size, and it is more rigorous than it sounds if done correctly rather than as a conversation.

1. 01Give them a task, not a tour. Find out whether this would suit you and buy it if it does. Then stop talking.
2. 02Ask them to think out loud, and resist helping. The silence where you want to explain something is the finding: that is the moment the page failed.
3. 03Watch what they do rather than listening to what they say about it. People are poor at reporting their own behaviour and excellent at demonstrating it.
4. 04Do it on their own phone, on their own connection, rather than on your machine where everything is cached.
5. 05Write down only the things that happened to more than one person. A single person's confusion is an anecdote; the same confusion twice is a defect.

Five people reliably surface the two or three problems that matter, and the same ones recur, which is how you know you have reached the end of the useful findings. A sixth person mostly confirms.

The objection is that this is not statistically valid, which is true and beside the point. You are not estimating a rate; you are finding faults, and faults do not need a sample size.