Machine Learning Foundations·Lesson 38 of 67·advanced
Choosing metrics that match the problem
Why accuracy lies on imbalanced data, what precision and recall actually trade off, and how to pick a threshold from the cost of each kind of mistake.
In this lesson · 20 min
- →Read a confusion matrix and compute precision, recall and F1 from it
- →Explain why accuracy is misleading when classes are imbalanced
- →Move the decision threshold deliberately, and see what it costs
- →Choose between ROC-AUC and precision-recall AUC
- →Pick regression metrics that match what you care about
A metric is a claim about what counts as good. Choosing the wrong one is how a model that optimises beautifully turns out to be useless — and the failure is invisible, because the number goes up either way.
Accuracy on imbalanced data
Both score around 0.97. One of them has never flagged anything.
The confusion matrix
Everything else is derived from four numbers:
The two questions are worth memorising in plain words:
- Precision — when the model says yes, how often is it right?
- Recall — of everything that really was a yes, how much did the model find?
Read the support column: the metrics for the rare class are computed from a handful of
examples, so they are noisy. macro avg treats both classes equally; weighted avg weights by
support and therefore looks almost exactly like accuracy.
The threshold
predict uses 0.5, but nothing makes 0.5 special. Moving it trades precision against recall:
Precision-recall and ROC curves
ROC-AUC looks impressive on imbalanced data because the false-positive rate has a huge denominator — a thousand extra false positives barely moves it. Average precision does not flatter you in the same way.
Regression metrics
Both have the same total absolute error. RMSE punishes the second far more, because squaring makes one big miss count for more than several small ones. Choose accordingly: RMSE when a large error is disproportionately bad, MAE when errors should count equally.
Try it yourself
From the confusion matrix counts, compute precision, recall and f1 by hand. Do not call scikit-learn's metric functions.
Find the lowest threshold from the list whose precision is at least 0.80, and put it in chosen. Set chosen_recall to the recall it achieves.
Check yourself
Your fraud detector scores 0.994 accuracy on data that is 0.6% fraud. What does that tell you?
Check yourself
A content filter must catch as much genuine abuse as possible; wrongly flagging a benign request is a minor annoyance. Which way do you move the threshold, and what do you buy?
Recap
- Accuracy is dominated by the majority class; on imbalanced data it says almost nothing.
- Everything comes from the confusion matrix. Precision: when it says yes, is it right. Recall: of everything that was a yes, how much did it find.
- The threshold is a product decision, driven by what each kind of error costs.
predictjust uses 0.5. - On imbalanced data prefer average precision over ROC-AUC — a random model’s AP is the positive rate, not 0.5.
- RMSE punishes a single large error more than MAE does; pick the one matching what actually hurts.
- Report metrics in units people can act on.