Controlled-experiment Using Pre-Experiment Data: a variance reduction method that adjusts each user's metric with a covariate measured before the test, usually the same metric from the pre-period. Less variance means narrower confidence intervals, so a test reaches the same power with fewer users.
Card 1 of 276. Answer: Controlled-experiment Using Pre-Experiment Data: a variance reduction method that adjusts each user's metric with a covariate measured before the test, usually the same metric from the pre-period. Less variance means narrower confidence intervals, so a test reaches the same power with fewer users.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The adjusted metric is Y − θ(X − mean of X), where X is the pre-experiment covariate and θ = cov(X, Y) / var(X), the slope from regressing Y on X. Because X was measured before assignment, the treatment can't affect it, so the adjustment leaves the estimated effect unbiased and only removes noise that X explains. Alex Deng, Ya Xu, Ron Kohavi and Toby Walker described it in 2013, from Bing's experimentation platform at Microsoft.
Example: the adjusted metric's variance is the original times 1 − ρ², where ρ is the correlation between X and Y. If last month's spend correlates 0.6 with spend during the test, variance falls by 36%, and since the required sample size scales with variance, the test needs about 64% of the users. With a correlation of 0.8, it needs only 36%.
Follow-ups: CUPED is equivalent to regression adjustment, as in ANCOVA; new users have no pre-period data, so they are usually given the covariate's mean, which leaves their metric unadjusted; and later variants such as DoorDash's CUPAC use a machine learning prediction of the outcome as the covariate to get a higher correlation.
