Watch proxies until the ground truth lands: data quality, drift in the inputs and in the predictions, and, for a well-calibrated classifier, an estimate of performance from its own confidence scores. Confirm with realized performance once the labels arrive.
Card 1 of 228. Answer: Watch proxies until the ground truth lands: data quality, drift in the inputs and in the predictions, and, for a well-calibrated classifier, an estimate of performance from its own confidence scores. Confirm with realized performance once the labels arrive.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Many models get their labels late: a loan defaults months after approval, churn is known 30 to 90 days on, fraud chargebacks take weeks. In that gap you can monitor data quality (missing values, schema, ranges), drift in the input features, drift in the prediction distribution, and an estimate of the model's performance.
Performance estimation uses the model's own probabilities. If a binary classifier is well calibrated, a prediction with score p is right with probability p when it predicts positive and 1 − p when it predicts negative. For four predictions scored 0.9, 0.8, 0.3 and 0.6, the expected accuracy is (0.9 + 0.8 + 0.7 + 0.6) / 4 = 0.75. If past batches estimated 0.85, something has changed before a single label is in. The open-source NannyML library implements this as confidence-based performance estimation (CBPE).
The catch: the estimate assumes the model's probabilities still mean what they meant in training. That holds under covariate shift but not under concept drift, where the model is confidently wrong in a way its scores can't reveal. So check calibration on recent labelled data, treat the estimate as an early warning, and compare it with realized performance as labels arrive.
