Module 4 · Data Analytics: Statistics & Data Visualization
A/B Testing: lessons
Designing, running and reading product experiments — including the mistakes that make the results meaningless.
4 lessons · 1h 3m
What this module teaches, what it asks of you first and how you practise it are on its syllabus page.
Open the syllabusDesigning an Experiment
16 minHypotheses, primary metrics, guardrail metrics and the unit of randomisation — the decisions to make before a single visitor enters the experiment.
Included with the track Simulation insideSample Size and Duration
15 minThe minimum detectable effect decides how many visitors you need, and the calendar decides how long — always in full weeks.
Included with the track Simulation insideAnalysing A/B Test Results
17 minA proportion test, a confidence interval for the difference, continuous guardrail metrics, and how to report the result to the people who have to decide.
Included with the track Simulation insideA/B Testing Pitfalls
15 minPeeking, sample ratio mismatch, novelty effects and too many metrics — four habits that feel reasonable and make a tidy experiment give the wrong answer.
Included with the track Simulation inside