A/B-Tests are a popular approach for testing the success of changes made in the process of optimising web offerings. The version with changes is delivered to some of the users, others see the unchanged original version. Then it is measured which version generates more conversions and after some time it is determined on a solid statistical basis which changes have been proven to increase coverage. This sounds as simple as it is desirable, but it can lead to deception and disappointment if the methodological basis is ignored.

The British data scientist Martin Goodson already explained in 2014 in a lecture(1) why many positive A/B-test results do not stand up to scientific scrutiny. The crux of the A/B-Testing is the observance of the Test-statistical power. Goodson argues convincingly how in still widespread practice many A/B-Tests with insufficient discriminatory power can lead to false positives not being detected and thus seemingly successful changes being implemented that at best have no effect, but at worst can even depress conversion figures.

How come?

Sample size

Goodson mentions 6000 conversions (in words: six thousand. Conversions, not visits) as the minimum size to be able to prove a performance improvement of 5% with sufficient statistical certainty. To prove an improvement of 10% - which is difficult to achieve in practice - 1600 conversions would still be necessary. Not every online offer can come up with such conversion figures within a reasonable period of time.

Test runtime

Many A/B-Tests are simply running too short. The results of the current Test are usually subject to ongoing monitoring. If the success of a variant becomes apparent, the Test ended quickly - often after only a few weeks or even days. Such an approach leads over several Tests almost certainly lead to false positive results.

The minimum duration of a A/B-Test can and should, like all parameters of the Test-Power, already in the run-up to the Tests be calculated statistically. An exemplary calculation via the Test-duration calculator from converlytics (2) gives for a simple A/B-Test with 1000 visitors per day and variant, with a conversion rate of 3% and an expected improvement of a solid 5%, a minimum duration of 204 days to reach a confidence level of 95%. At 100 visitors per day and variant, it's 2031 days (yes, that's actually a good five and a half years).

Too many Tests

Test, test, test, then success will come quickly. This is a widespread maxim in the use of A/B-Tests and the appearance of success is indeed usually created with this approach. In his lecture, Martin Goodson shows the statistical background of this typical and yet false observation: with increasing numbers of Tests The number of false-positive results, the lack of Test-Power leads to them not being recognised as such.

What to do?

The idea of the A/B-The use of the tool of testing is by no means obsolete with Goodson's remarks. However, it goes without saying that one should know how to do it correctly and whether it is suitable as a tool for one's own strategy. Careful preparation and attention to statistical test quality are essential prerequisites for useful results.

For small and medium-sized online offerings, however, this can mean that A/B-Tests are not economically and statistically efficient to use, as the necessary visitor and conversion figures cannot be achieved in a reasonable amount of time.

If in such a case A/B-Tests If, at the price of statistical power, too small samples and too short durations are implemented and the successes that undoubtedly occur are only a sham, this is a waste of resources that would be better spent on other, precisely non-test-driven, areas of marketing optimisation.

To quote Martin Goodson himself:

"I don't think you can trust the data for everything. You should trust
your experience and your education, because people know how to sell
things to other people, they've been doing it for thousands of years."

Inspiring conclusion of a lecture that was as well-founded as it was sobering.

1: https://www.youtube.com/watch?v=MdkHLS0FPMk
2: https://converlytics.com/dauer-calculator-ab-test/