A model can be accurate and still lie about its confidence

Accuracy asks whether the top class is right. Calibration asks whether a predicted probability of 0.8 means the model is right about 80 percent of the time. A model can be excellent at the first and badly wrong about the second, and modern deep networks usually are: they are systematically overconfident. Guo et al. is the reference, and the uncomfortable finding is that overconfidence got worse as networks got better, not better.

This bites hardest when a downstream decision reads the probability rather than the label. Anything with a review threshold, a routing rule, or an abstain option is consuming the number, not the argmax. An overconfident model set to "escalate anything below 0.7 confidence" will escalate almost nothing, and the errors it keeps will be the confident ones.

The fix is smaller than people expect. Temperature scaling divides the logits by a single scalar learned on a validation set. One parameter. Because dividing every logit by the same positive number is monotonic, the ranking of classes does not change, which means accuracy is mathematically unchanged. You get better probabilities for free.

Practical notes:

• Fit the temperature on held-out data, never on the training set.
• Check with a reliability diagram, not a single number. Expected calibration error hides where the miscalibration lives, and it is usually not uniform across the range.
• Recheck after any distribution shift. Calibration is a property of the model and the data together, so it does not survive a change in the input mix.

The reason I keep coming back to this: when someone says a model is "90 percent confident," the interesting question is whether anyone has ever checked what that number does in aggregate. Often nobody has.