Running a new model next to the live one on real production traffic, logging its predictions without acting on them. It tests the model on real inputs, latency and load with no effect on users.
Card 1 of 228. Answer: Running a new model next to the live one on real production traffic, logging its predictions without acting on them. It tests the model on real inputs, latency and load with no effect on users.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
In a shadow deployment (or shadow mode) a copy of every request, or of a sample of them, goes to the candidate as well as the live model. Only the live model's output is used; the candidate's predictions are logged and compared, first with the live model's and later with the true outcomes when they arrive.
Example: a bank replacing its fraud model runs the new one in shadow for two weeks on 3 million transactions. It learns that the two models disagree on 4% of transactions, mostly small online purchases; that the candidate's 99th-percentile latency is 180 ms against a 100 ms budget; and that one of its features is always null for in-store payments. All three surface before any customer is affected.
Its limits: shadowing can't measure effects that depend on acting on the predictions, such as clicks on recommendations, so that needs a canary release or an A/B test. It doubles inference cost while it runs. And the outcomes it is judged on were shaped by the live model; in fraud, only transactions the live model approved ever get a label. A common order is offline evaluation, shadow, canary, A/B test, full rollout.
