The probability, if the null hypothesis were true, of getting a result at least as extreme as the one observed. A small p-value means the data would be surprising if there were no effect; it is not the probability that the null hypothesis is true.
Card 1 of 244. Answer: The probability, if the null hypothesis were true, of getting a result at least as extreme as the one observed. A small p-value means the data would be surprising if there were no effect; it is not the probability that the null hypothesis is true.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
A p-value measures how compatible the data are with the null hypothesis. You pick a test statistic, work out how it would be distributed if the null were true, and ask how often a value at least as extreme as yours would turn up. If that probability is below the significance level α, often 0.05, the result is called statistically significant and the null is rejected.
Example: a coin lands heads 60 times in 100 flips. If it is fair, the chance of 60 or more heads is about 2.8%, and of a result at least that far from 50 in either direction about 5.7%. The two-sided p-value is 0.057, so the result is not significant at 0.05, even though the coin may well be biased. Absence of significance is not evidence of fairness.
Interviewers probe the misreadings. A p-value is not the probability that the null is true or that the result is a fluke, and it says nothing about the size of the effect: with a million users, a 0.1% lift can have p < 0.001. The American Statistical Association's 2016 statement on p-values warns against basing conclusions on whether p crosses 0.05 alone, so report the effect size and a confidence interval with it.
