The probability that a test rejects the null hypothesis when a real effect of a given size exists, 1 − β. It grows with sample size, effect size and the significance level; 80% is the usual target when planning an experiment.
Card 1 of 244. Answer: The probability that a test rejects the null hypothesis when a real effect of a given size exists, 1 − β. It grows with sample size, effect size and the significance level; 80% is the usual target when planning an experiment.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Power is the chance of detecting an effect that is really there. A power analysis fixes three of four quantities, the significance level α, the power, the smallest effect worth detecting and the sample size, and solves for the fourth, usually the sample size, before the experiment starts.
Example: an A/B test on a 10% conversion rate wants to detect a lift to 11% with α = 0.05, two-sided, and 80% power. The standard formula for two proportions gives about 14,800 users per arm. To detect a lift to 10.5% instead takes about 58,000 per arm: halving the effect roughly quadruples the sample, because the required size grows with 1 / effect².
Low power does more damage than missed effects. When an underpowered test does come out significant, the estimated effect is likely exaggerated, the "winner's curse" that Gelman and Carlin called a Type M error. Power computed after the fact from the observed effect adds nothing to the p-value. And stopping a test as soon as it looks significant, "peeking", inflates false positives unless the design allows for it.
