Define a proper scoring rule. Show that squared error on a probability, the Brier score, is proper by finding the p that minimises the expected score when the true probability is q, and give one commonly used score that is not proper.

Define a proper scoring rule. Show that squared error on a probability, the Brier score, is proper by finding the p that minimises the expected score when the true probability is q, and give one commonly used score that is not proper.

Approach: Write the expected score as a function of the reported probability with the true probability held fixed, differentiate, and check where the minimum sits.

A scoring rule is proper when the forecaster's expected score is optimised by reporting the true probability, and the Brier score is proper because E[S] = q(1 - p)^2 + (1 - q)p^2 has derivative -2q(1 - p) + 2(1 - q)p = 2(p - q), which is zero only at p = q and the second derivative is 2 > 0. Log loss is also proper: E[S] = -q log p - (1 - q) log(1 - p) has derivative -q/p + (1 - q)/(1 - p), again zero only at p = q. Accuracy after thresholding at 0.5 is not proper, because any report on the same side of the threshold as q scores identically, so a forecaster who believes 0.51 is free to report 0.99 and is never penalised for the exaggeration. That is why accuracy cannot be used to select between probabilistic models that will be sized on. Log loss penalises confident errors without bound, which makes it sensitive to a single mislabelled row, while the Brier score is bounded and treats a confident error as at most 1. On a desk the choice follows the use: log loss when the probability feeds a Kelly-style sizing that blows up near the extremes, Brier when a handful of bad labels is expected in the data.

Follow-up: Is the ranking-based AUC a proper scoring rule for the probabilities, and what does that imply about selecting a model on AUC and then sizing on its output?

Key concepts: proper scoring rule, Brier score, expected score, log loss.