A classifier on 10,000 samples gives TP = 90, FP = 210, FN = 30, TN = 9670. Compute precision, recall, F1 and accuracy to three decimals, and state what the accuracy figure hides.

A classifier on 10,000 samples gives TP = 90, FP = 210, FN = 30, TN = 9670. Compute precision, recall, F1 and accuracy to three decimals, and state what the accuracy figure hides.

Approach: Apply the definitions from the four cells, then compare the accuracy with the accuracy of the always-negative classifier on the same class balance.

Precision 0.300, recall 0.750, F1 0.429, accuracy 0.976. Precision is TP/(TP + FP) = 90/300 = 0.300, recall is TP/(TP + FN) = 90/120 = 0.750, the F1 score is the harmonic mean 2*0.3*0.75/(0.3 + 0.75) = 0.45/1.05 = 0.4286, and accuracy is (90 + 9670)/10000 = 0.976. The positive class is 120 of 10,000, that is 1.2%, so the classifier that predicts negative on every row scores 98.8% accuracy and beats this model on accuracy while having zero recall. The 97.6% figure is therefore worse than the trivial baseline, which the accuracy number alone does not show. The F1 score of 0.429 is the honest headline because it ignores the 9,670 true negatives entirely. The precision of 0.300 says two of every three alerts are wrong, which is the number that decides whether the model is usable when acting on a positive costs money.

Follow-up: If acting on a false positive costs $1 and a true positive earns $8, what is the expected profit per 10,000 rows here, and at what precision does it break even?

Key concepts: precision, recall, F1 score, class imbalance.