Offline metrics are computed on logged data that the old system shaped, against a proxy target, while the A/B test measures the real outcome on live users. Selection bias in the logs, a weak proxy, feedback loops and added latency all open a gap.
Card 1 of 228. Answer: Offline metrics are computed on logged data that the old system shaped, against a proxy target, while the A/B test measures the real outcome on live users. Selection bias in the logs, a weak proxy, feedback loops and added latency all open a gap.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
An offline evaluation scores a model on historical data with a metric such as AUC or NDCG. An online A/B test randomizes users between the old and new model and measures what the business cares about, such as revenue, retention or watch time. The two disagree more often than people expect.
Example: a new ranking model lifts NDCG by 3% on last month's click logs but lowers revenue per session in the A/B test. The logs only contain items the old model chose to show, so the new model is graded on a biased sample. Clicks reward eye-catching items while revenue needs purchases, so the proxy was wrong. And the larger model added 60 ms to every page, which costs engagement on its own.
What helps: track how well offline gains have predicted online gains across past launches, and trust a metric only as far as that record goes; use counterfactual estimators such as inverse propensity scoring on logs that record how likely each item was to be shown; use interleaving to compare rankers with less traffic; and keep the A/B test as the final word.
