A fix for multiple comparisons: when you run m tests, compare each p-value with α/m instead of α. It keeps the probability of any false positive, the family-wise error rate, at or below α, at the cost of power.
Card 1 of 244. Answer: A fix for multiple comparisons: when you run m tests, compare each p-value with α/m instead of α. It keeps the probability of any false positive, the family-wise error rate, at or below α, at the cost of power.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Every extra test is another chance of a false positive. With 20 independent tests of true null hypotheses at α = 0.05, the chance that at least one comes out significant is 1 − 0.95²⁰ ≈ 64%. Bonferroni divides α by the number of tests, which bounds the family-wise error rate at α whether or not the tests are independent.
Example: an A/B test reports 10 metrics. Without a correction, the chance that at least one moves "significantly" by luck alone is about 40% if the tests are independent. Bonferroni tests each metric at 0.05/10 = 0.005, so a metric with p = 0.01 no longer counts.
Its weakness is that it is conservative, more so when the tests are correlated, so real effects get missed. Holm's step-down method (1979) controls the same error rate and always rejects at least as many hypotheses. When screening many hypotheses, as in genomics or feature selection, the Benjamini-Hochberg procedure (1995) controls the false discovery rate instead, the expected share of false positives among the discoveries, and keeps far more power.
