https://datatweets.com/courses/ab-testing/designing-an-experiment/Recent content in Designing an Experiment on DATATWEETSHugoenCopyright (c) 2025 DatatweetsSun, 28 Jun 2026 09:00:00 +0200https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-1-from-question-to-hypothesis/Fri, 20 Feb 2026 09:00:00 +0200https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-1-from-question-to-hypothesis/An experiment can only answer a question that’s sharp enough to be wrong. This lesson turns a fuzzy product goal into a testable hypothesis — specific, directional, falsifiable — and introduces the null and alternative hypotheses that frame every A/B decision.https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-2-choosing-your-metrics/Fri, 20 Feb 2026 09:00:00 +0200https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-2-choosing-your-metrics/Every experiment needs a metric hierarchy: one primary metric that drives the decision, guardrail metrics that must not get worse, and secondary metrics for context. This lesson defines each tier, lays out what makes a good primary metric, and computes real metrics from seeded Lumen data so you can see conversion, ARPU, and a p95 latency guardrail side by side.https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-3-the-randomization-unit/Fri, 20 Feb 2026 09:00:00 +0200https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-3-the-randomization-unit/The randomization unit is what you flip the coin on, and it’s easy to get wrong. This lesson compares randomizing by user, by session, and by cluster; explains the two properties the unit must satisfy — consistency and independence; and shows how a deterministic hash gives each user a stable assignment across every visit.https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-4-the-minimum-detectable-effect/Fri, 20 Feb 2026 09:00:00 +0200https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-4-the-minimum-detectable-effect/Not every real effect is worth shipping, and not every experiment can afford to chase a tiny one. This lesson defines the minimum detectable effect (MDE) as a design choice grounded in practical significance, separates absolute from relative effects, and shows — with real scipy code — how a smaller MDE explodes the sample size you need.https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-5-guided-project-design-lumens-pricing-experiment/Fri, 20 Feb 2026 09:00:00 +0200https://datatweets.com/courses/ab-testing/designing-an-experiment/lesson-5-guided-project-design-lumens-pricing-experiment/Put Module 2 together into one design doc for a real Lumen experiment. You’ll write a specific hypothesis, choose primary, guardrail, and secondary metrics, pick a randomization unit, set an MDE, and compute the exact sample size with scipy — the full contract you fix before a single user is bucketed.