Welcome to Analyzing Proportion Metrics, the fourth module. You’ve designed an experiment and sized it. Now the data is in — and it’s time to answer the question the whole thing was built for: did the change work? Most A/B tests hinge on a proportion — a conversion rate, a signup rate, a click-through rate — and comparing two of them is the single most common analysis in experimentation. This module teaches you to do it properly, and to say exactly what the result does and doesn’t mean.

You’ll start with the two-proportion z-test: how a difference in rates becomes a test statistic and a p-value. You’ll learn to read a p-value without the misinterpretations that trip up even experienced teams. You’ll build a confidence interval for the difference — which tells you not just whether there’s an effect but how big it plausibly is — and see how it and the p-value are two views of the same evidence. Finally you’ll combine significance, the interval, practical significance, and guardrails into an actual ship/no-ship decision. The module analyzes Lumen’s signup experiment — the same one you designed and sized — from raw counts to a verdict.

Every number here is real, runnable Python: the z-test, p-value, and confidence interval are computed with numpy and scipy on the exact seeded experiment from earlier modules, so the analysis connects design to sizing to result. Start with Lesson 1 on the two-proportion z-test.