Course

A/B Testing & Experimentation

Design, size, and analyze controlled experiments the way modern product teams do — randomization, power, proportion and mean tests, and the pitfalls that invalidate results, hands-on in Python.

At a glance

Level
Intermediate
Lessons
10 lessons across 2 modules
What you build
A full experiment, designed and analyzed
Cost
100% free · no API key needed

What you'll build

You'll run experiments for Lumen, a fictional e-learning app, across the whole course — from the logic of a controlled experiment to designing one, sizing it, analyzing conversion and revenue metrics, avoiding the classic pitfalls, and reading out a decision. Every module ends with a guided project, and the course closes with a capstone where you design, simulate, and analyze a complete experiment. All the data is seeded synthetic and generated in-course, and every statistic is computed with real numpy, scipy, and statsmodels — fully reproducible, no API key, no cost.

Course syllabus

Work through the modules at your own pace. Each lesson is a self-contained, hands-on read.

1 The Logic of Experiments 5 lessons · 1 week
2 Designing an Experiment 5 lessons · 1 week

Before you start

You'll need comfortable Python — functions, arrays, and basic numpy — and a first exposure to statistics: sampling, the normal distribution, and the idea of a hypothesis test. Our Statistics & Probability course is the ideal prerequisite: it covers sampling, distributions, and hypothesis-testing basics, which this course builds directly on to design and analyze real experiments. If you're new to Python, start with Python for Data Analytics.

Set up your environment

You can complete this course on any machine with Python 3.12+. There's no API key and nothing to pay for — every experiment runs locally on seeded synthetic data.

  1. Install the packages the course uses:
pip install numpy scipy statsmodels

That's it. Every dataset is generated in the lesson with a fixed random seed, so your numbers will match the ones shown exactly, and you can rerun any analysis end to end.

Package APIs shift over time. If a function signature has changed since these versions, the statistical reasoning still applies — adjust the syntax to what you have installed.

Ready to run your first experiment?

Start with the logic of experiments and work through every module, from why we randomize to reading out a real decision.

Start the first lesson