Skip to content

Machine Learning Foundations·Lesson 38 of 67·advanced

Choosing metrics that match the problem

Why accuracy lies on imbalanced data, what precision and recall actually trade off, and how to pick a threshold from the cost of each kind of mistake.


In this lesson · 20 min

  • →Read a confusion matrix and compute precision, recall and F1 from it
  • →Explain why accuracy is misleading when classes are imbalanced
  • →Move the decision threshold deliberately, and see what it costs
  • →Choose between ROC-AUC and precision-recall AUC
  • →Pick regression metrics that match what you care about

A metric is a claim about what counts as good. Choosing the wrong one is how a model that optimises beautifully turns out to be useless — and the failure is invisible, because the number goes up either way.

Accuracy on imbalanced data

PythonPython not started

Both score around 0.97. One of them has never flagged anything.

The confusion matrix

Everything else is derived from four numbers:

PythonPython not started
PythonPython not started

The two questions are worth memorising in plain words:

  • Precision — when the model says yes, how often is it right?
  • Recall — of everything that really was a yes, how much did the model find?
PythonPython not started

Read the support column: the metrics for the rare class are computed from a handful of examples, so they are noisy. macro avg treats both classes equally; weighted avg weights by support and therefore looks almost exactly like accuracy.

The threshold

predict uses 0.5, but nothing makes 0.5 special. Moving it trades precision against recall:

PythonPython not started

Precision-recall and ROC curves

PythonPython not started

ROC-AUC looks impressive on imbalanced data because the false-positive rate has a huge denominator — a thousand extra false positives barely moves it. Average precision does not flatter you in the same way.

PythonPython not started

Regression metrics

PythonPython not started

Both have the same total absolute error. RMSE punishes the second far more, because squaring makes one big miss count for more than several small ones. Choose accordingly: RMSE when a large error is disproportionately bad, MAE when errors should count equally.

Try it yourself

Exercise 1

From the confusion matrix counts, compute precision, recall and f1 by hand. Do not call scikit-learn's metric functions.

Your turnPython not started
Exercise 2

Find the lowest threshold from the list whose precision is at least 0.80, and put it in chosen. Set chosen_recall to the recall it achieves.

Your turnPython not started

Check yourself

Your fraud detector scores 0.994 accuracy on data that is 0.6% fraud. What does that tell you?

Check yourself

A content filter must catch as much genuine abuse as possible; wrongly flagging a benign request is a minor annoyance. Which way do you move the threshold, and what do you buy?

Recap

  • Accuracy is dominated by the majority class; on imbalanced data it says almost nothing.
  • Everything comes from the confusion matrix. Precision: when it says yes, is it right. Recall: of everything that was a yes, how much did it find.
  • The threshold is a product decision, driven by what each kind of error costs. predict just uses 0.5.
  • On imbalanced data prefer average precision over ROC-AUC — a random model’s AP is the positive rate, not 0.5.
  • RMSE punishes a single large error more than MAE does; pick the one matching what actually hurts.
  • Report metrics in units people can act on.