· intermediate
A/B Testing and Experimentation
How to run a test whose answer means something. What you are actually assuming, why stopping early breaks it, how big the sample has to be before you start, and what a p-value does and does not tell you.
Written in house, from the work rather than from a syllabus. Every example is one we would use ourselves, and every quiz question explains why its answer is right.
What you’ll learn
- State a hypothesis and the metric that would settle it, before collecting data
- Explain what randomisation buys and what it cannot fix
- Work out the sample size a test needs from the effect worth detecting
- Read a p-value and a confidence interval for what each actually claims
- Recognise peeking and multiple comparisons, and why both inflate false positives
- Separate a statistically detectable effect from one worth acting on
Before you start
- · Comfort with averages, percentages and a little probability
- · No prior experiment design