A model predicting a rare event with base rate 0.5% reports 99.4% accuracy and ROC-AUC of 0.86. Explain why the accuracy figure is uninformative, and say whether the AUC or the precision-recall curve better describes the model here.

A model predicting a rare event with base rate 0.5% reports 99.4% accuracy and ROC-AUC of 0.86. Explain why the accuracy figure is uninformative, and say whether the AUC or the precision-recall curve better describes the model here.

Approach: Compare each metric with what the trivial classifier achieves, then track how the false positive rate and precision move when the negative class is 200 times the positive one.

The precision-recall curve is the informative one, because at a 0.5% base rate the always-negative classifier already scores 99.5% accuracy and the ROC curve's false positive rate axis is diluted by the huge negative class. Accuracy at this balance is a statement about the majority class and the model is beaten by predicting no event every time. ROC-AUC is invariant to the class ratio, which is presented as a strength and is the problem here: a false positive rate of 1% sounds small but corresponds to 0.01 * 199 = 1.99 false positives per true positive available, so precision at that operating point is at most 0.33 even with perfect recall. The precision-recall curve makes that visible because its baseline is the base rate 0.005 rather than the diagonal, so an AUC-PR of 0.15 is a thirty-fold lift while an ROC-AUC of 0.86 could correspond to a model with no usable operating point at all. For a trading signal the right report is precision at the recall the capacity allows, or expected profit against a stated cost matrix, since the desk can only act on a bounded number of alerts a day.

Follow-up: If the base rate falls from 0.5% to 0.1% while the score distributions per class are unchanged, what happens to ROC-AUC and to average precision?

Key concepts: base rate, false positive rate, precision-recall curve, ROC-AUC.