A model outputs probabilities and is well ranked but badly calibrated: events it calls 70% happen 45% of the time. Explain what this costs on a sized trade, how you would measure the miscalibration…

Question

A model outputs probabilities and is well ranked but badly calibrated: events it calls 70% happen 45% of the time. Explain what this costs on a sized trade, how you would measure the miscalibration, and how Platt scaling and isotonic regression differ as fixes.

Accepted Answer

Miscalibration does not change the ranking or the AUC, but it directly mis-sizes any position whose size is a function of the probability, and the fix is a monotone recalibration fitted on held-out data, Platt scaling when data is scarce and isotonic regression when it is not. If size is proportional to edge, believing 70% when the truth is 45% turns a position that should be small or absent into a large one, and under a Kelly rule the overbet is quadratic in the error, so the growth rate can go negative while the classifier still ranks correctly. Measure it with a reliability diagram: bin the predictions, plot observed frequency against mean predicted probability, and read the deviation from the diagonal, summarised as expected calibration error or as the calibration component of the Brier score decomposition. Platt scaling fits a one-parameter-plus-intercept logistic on the model score, so it imposes a sigmoid shape and needs few points, which suits a few hundred validation rows. Isotonic regression fits any non-decreasing step function, so it corrects arbitrary distortions but needs more data and will overfit on small samples, producing flat regions that collapse distinct scores. Both are monotone, so neither can improve AUC, and both must be fitted on data disjoint from training and refitted when the regime changes.