Trying many analyses, such as different metrics, subgroups, outlier rules or stopping points, and reporting only those that reach p < 0.05. Each extra try is another chance for a false positive, so the reported p-value overstates the evidence.
Card 1 of 276. Answer: Trying many analyses, such as different metrics, subgroups, outlier rules or stopping points, and reporting only those that reach p < 0.05. Each extra try is another chance for a false positive, so the reported p-value overstates the evidence.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
A p-value is valid for one analysis decided in advance. When the analysis is chosen after seeing the data, the real false positive rate is much higher than the nominal 5%, even if no single step looks dishonest. Gelman and Loken called this the garden of forking paths: many reasonable choices, each of which could have gone another way.
Example: test 20 independent metrics in an experiment with no real effect, and the chance that at least one comes out significant is 1 − 0.95²⁰ ≈ 64%. Simmons, Nelson and Simonsohn showed in 2011 that combining four common flexible choices, such as adding a covariate or collecting more data after a first look, pushed the false positive rate in their simulations to about 61%.
Fixes interviewers expect: decide the primary metric, sample size and analysis before the test, or pre-register them; correct for multiple comparisons; don't stop an A/B test the first time it looks significant unless you use a sequential method built for repeated looks; and report every analysis you ran, including the ones that didn't work.
