That the result would be unlikely if there were no real effect: its p-value is below a threshold, the significance level α, chosen before the test, usually 0.05. It says the effect is probably not zero, not that it is large or important.
Card 1 of 276. Answer: That the result would be unlikely if there were no real effect: its p-value is below a threshold, the significance level α, chosen before the test, usually 0.05. It says the effect is probably not zero, not that it is large or important.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The significance level α is the false positive rate you accept: if there is no effect, a test at α = 0.05 will still call 5% of results significant. Whether a result clears the bar depends on the effect's size, the noise in the data and the sample size, so significance is not a property of the effect alone. Practical significance asks a different question: is the effect large enough to matter for the decision?
Example: with a million users in each arm of an A/B test, a conversion rate moving from 10.00% to 10.10% gives z ≈ 2.35 and p ≈ 0.019, significant but perhaps too small to be worth shipping. With 200 users per arm, a move from 10% to 12% gives z ≈ 0.64 and p ≈ 0.52, not significant even though the lift would matter if it were real.
Fisher suggested 0.05 as a convenient cut-off in 1925, and it stuck. The American Statistical Association's 2016 statement on p-values warned against treating that line as proof, and in 2018 Benjamin and colleagues proposed 0.005 for new discoveries. In interviews, pair significance with an effect size and a confidence interval.
