Its predicted probabilities match observed frequencies: of all the cases it scores 0.8, about 80% turn out positive. A calibration curve, or reliability diagram, plots the observed positive rate against the predicted probability; a calibrated model follows the diagonal.
Card 1 of 259. Answer: Its predicted probabilities match observed frequencies: of all the cases it scores 0.8, about 80% turn out positive. A calibration curve, or reliability diagram, plots the observed positive rate against the predicted probability; a calibrated model follows the diagonal.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Calibration is about whether a probability means what it says. It is separate from discrimination, which AUC measures: a model can rank cases perfectly and still output 0.9 for cases that are positive only 60% of the time. Calibration matters whenever the probability itself drives a decision, such as an expected cost, a price or a risk threshold.
To draw the curve, bin the predictions, for example into tenths, and plot each bin's mean predicted probability against the share of positives in it. If cases scored 0.7 to 0.8 are positive only 55% of the time, the model is overconfident there. The expected calibration error averages these gaps across bins, weighted by how many predictions each bin holds.
Some models drift from the diagonal in typical ways: naive Bayes pushes probabilities towards 0 and 1, while SVMs and boosted trees push them away from the extremes (Niculescu-Mizil and Caruana, 2005), and Guo et al. (2017) found modern deep networks overconfident. Undersampling the majority class also shifts probabilities. Fix it on held-out data with Platt scaling, isotonic regression or, for neural networks, temperature scaling. The Brier score and log loss both reward calibration.
