Skip to content

Evaluation Metrics Deep Dive

What This Is

Metric choice decides what "better" means. A model can win on accuracy and fail the real job. A model can rank well and still have useless probabilities. A model can post a high F1 and still waste the reviewer budget. This page is about one decision:

  • which metric should control model choice for this task

That decision has to match the consequence of the mistake, not a default from the textbook.

When You Use It

  • choosing the primary model-selection metric
  • comparing models that win on different numbers
  • explaining results to a team with a specific operational cost
  • deciding whether ranking, thresholding, or probability quality matters most
  • picking between macro and micro averaging on a multi-class task
  • reporting a metric at a constrained review budget (top-k)

Do Not Use It When

  • the split itself is unfair — see Honest Splits and Baselines. Metrics on a leaky split are theater.
  • the deployment threshold is not yet decided and you are forced to report a scalar — report the operating curve instead.

Start With The Decision Type

Choose the metric by what the score will actually control:

Need Better first metric Why
balanced classes, equal costs accuracy simple, but only when the class mix is honest
rare positive class average precision or balanced accuracy accuracy hides minority-class failure
ranking before threshold choice ROC AUC or average precision the score orders cases rather than making one hard call
hard threshold policy precision, recall, or cost at the chosen cutoff the threshold is the decision
trustworthy probabilities log loss or Brier score probability quality matters, not just ranking
fixed review budget precision@k or recall@k only the top-k items are ever acted on
multi-class, many classes macro-F1 or balanced accuracy per-class weight, not per-sample
multi-class, class prior matters micro-F1 or accuracy samples count, not classes

Quick Rule

Ask three questions:

  1. is the class balance skewed
  2. is one error worse than the other
  3. does the score control ranking, thresholding, or calibrated probability

If you cannot answer those, the metric is still arbitrary.

Read The Confusion Matrix First

For a binary classifier at a fixed threshold, many decision metrics begin with the same four cells:

                   predicted +          predicted -
actual +    True Positive  (TP)   False Negative (FN)
actual -    False Positive (FP)   True Negative  (TN)

From those four numbers you can compute thresholded metrics such as:

  • accuracy = (TP + TN) / all
  • precision = TP / (TP + FP) — of the items I flagged, how many were right
  • recall (TPR, sensitivity) = TP / (TP + FN) — of the real positives, how many did I catch
  • specificity (TNR) = TN / (TN + FP) — of the real negatives, how many did I leave alone
  • F1 = harmonic mean of precision and recall — penalizes imbalance between them
  • FPR = FP / (FP + TN) — the x-axis of the ROC curve
  • balanced accuracy = (recall + specificity) / 2 — class-balanced in one number

Workflow: look at the raw 2×2 counts before computing a scalar. A 99% accuracy on a 1% positive problem where every positive is missed is easy to spot in the matrix and invisible in the scalar.

Cost-Matrix Framing

When errors have different real-world costs, bolt the costs onto the matrix and optimize the expected cost, not a default metric:

                   predicted +   predicted -
actual +              0             c_FN
actual -              c_FP           0

Expected cost = c_FP · P(FP) + c_FN · P(FN), plus any costs assigned to correct decisions. On a finite validation set, evaluate candidate thresholds and choose a minimizer; several thresholds can tie. Typical illustrative asymmetries (which must be confirmed for the real deployment) include:

  • medical screening: c_FN >> c_FP (missing a sick patient is worse than a follow-up visit)
  • content moderation: c_FP >> c_FN (a wrong takedown is worse than a missed borderline post)
  • spam filter at inbox: c_FP (legit mail in spam) >> c_FN (spam in inbox)

The decision becomes: define costs with stakeholders, pick a threshold on held-out predictions, and report both cost and the confusion matrix there. Calibration is not required to sweep empirical thresholds, but calibrated probabilities matter when a probability itself drives expected-utility decisions or thresholds must transfer across populations. See Calibration and Thresholds.

Ranking Metrics — When The Threshold Comes Later

If the eventual consumer ranks cases and triages the top-k, the right metric lives on the ranking, not on a single cutoff:

  • ROC AUC — probability a random positive outranks a random negative. Threshold-free, but insensitive to class prior — on a 1% positive task, a model with AUC 0.95 can still be useless at the operating point.
  • average precision (AP) — area under precision-recall curve. Much more honest than AUC on imbalanced tasks because it weighs the early-ranked region where the action happens.
  • precision@k — of the top-k ranked items, how many are truly positive. The correct metric when the reviewer can handle exactly k per day.
  • recall@k — of all real positives, how many are in the top-k. The correct metric when you need to catch most of them inside a budget.
  • NDCG — for ordered lists where position matters with a decay (search, recommenders).

Practical rule: if the downstream user ever types a number k (top 100 leads, top 50 suspicious transactions, top 200 alerts), the evaluation number should end in @k.

Multi-Class: Macro vs. Micro vs. Weighted

For more than two classes, averaging choices change the story:

Average Formula sketch When it matches the job
macro mean of per-class F1 you care that every class works — minority classes count the same as majority
micro pool TP/FP/FN across all classes, then compute one global precision and recall, take their harmonic mean — equals accuracy for single-label tasks you care about overall per-sample correctness, majority classes dominate
weighted per-class F1 weighted by support you want per-class performance weighted by real class prior

A model that scores 0.9 macro-F1 and 0.95 micro-F1 is honest about all classes. A model that scores 0.65 macro-F1 and 0.95 micro-F1 is hiding minority-class failure behind a frequent-class win. Report both on any multi-class task; they catch different problems.

For imbalanced multi-class problems (rare disease categories, imbalanced taxonomies), macro is usually the honest primary and micro the secondary.

Minimal Pattern

from sklearn.metrics import (
    confusion_matrix,
    classification_report,
    accuracy_score, balanced_accuracy_score,
    precision_score, recall_score, f1_score,
    average_precision_score, roc_auc_score,
    brier_score_loss, log_loss,
)

# inspect the matrix before the scalars
print(confusion_matrix(y_true, y_pred))
print(classification_report(y_true, y_pred, digits=3))

# ranking metrics on scores, not labels
ap  = average_precision_score(y_true, y_score)
auc = roc_auc_score(y_true, y_score)

# calibration check, only when probabilities matter
brier = brier_score_loss(y_true, y_prob)
ll    = log_loss(y_true, y_prob)

# precision@k — the right metric for fixed review budgets
import numpy as np
k = 100
scores = np.asarray(y_score)
labels = np.asarray(y_true)
top_idx = np.argsort(-scores)[:k]
precision_at_k = labels[top_idx].mean()

The important move is not computing every metric. The important move is deciding which one should be primary and why.

What To Inspect

  • the confusion matrix at the chosen threshold — raw counts, not only the scalar
  • class balance in the held-out set — not just training
  • the precision-recall curve, not just AUC
  • whether two models swap places under different metrics
  • whether the chosen metric actually matches the downstream decision
  • per-class F1 on multi-class tasks — the macro average hides which class is broken
  • stability of the metric across CV folds — a metric that wobbles more than the model gap is not a decision tool

If the primary metric changes, the story about the "best" model can change with it. That is the signal to stop and re-read the task.

Failure Pattern

The classic failure is choosing accuracy on a rare-event task. A 99%-negative dataset produces a 99%-accurate model that does nothing — the confusion matrix catches it; the scalar does not.

The second classic failure is ROC AUC on an imbalanced task with a review budget. AUC 0.95 can coexist with precision@100 = 0.10 when positives are rare — the model ranks well in general but not where the action happens.

The third failure is macro vs. micro confusion on multi-class: reporting micro-F1 on an imbalanced taxonomy and declaring success while every rare class is broken.

Common Mistakes

  • reporting one metric as if it tells the whole story
  • using accuracy on imbalanced data
  • calling ROC AUC proof of good probabilities (it is not — AUC is rank quality; Brier/log-loss is probability quality)
  • optimizing F1 without saying whether precision or recall matters more
  • comparing scores across datasets with different class balance and acting as if they mean the same thing
  • choosing macro-F1 when the deployment cares about per-sample correctness, or micro-F1 when it cares about per-class coverage
  • reporting precision@k or recall@k without saying what k is
  • changing the primary metric after seeing the numbers ("metric shopping")

A Good Metric Note

After one experiment, the learner should be able to say:

  • which metric is primary and why it matches the task
  • which secondary metric guards against a blind spot
  • what the confusion matrix looks like at the operating threshold
  • what decision would change if the metric changed

If any of those four are missing, the metric is decoration, not a decision.

Decision: Which Metric Is Primary

Task shape Primary Secondary Why
balanced binary, no cost asymmetry accuracy confusion matrix sanity
rare-event binary, ranking use average precision recall@k rank + operational budget
rare-event binary, hard threshold expected cost @ threshold precision, recall cost drives the cut
probabilistic output consumed downstream Brier score log loss, reliability diagram calibration matters
multi-class, balanced accuracy / micro-F1 macro-F1 majority agrees with per-class
multi-class, imbalanced macro-F1 per-class F1 table minority class visibility
ordered list output NDCG@k precision@k position-weighted

Practice

  1. Take a rare-event dataset (e.g., 2% positive fraud). Build one baseline. Report accuracy, balanced accuracy, AP, AUC, precision@100, recall@100. Explain which is primary and why.
  2. Produce a confusion matrix at three thresholds — default 0.5, a recall-80% threshold, and a precision-80% threshold. Narrate the shifts in TP/FP/FN/TN.
  3. Define a cost matrix with c_FN = 5·c_FP. Find the threshold that minimizes expected cost on validation. Report the cost at that threshold versus the default 0.5.
  4. On a multi-class task, compute macro-F1, micro-F1, and weighted-F1. Find a case where they disagree sharply and explain the disagreement from the per-class F1 table.
  5. Train two models that swap on AUC vs. AP. Decide which is better for a review-budget deployment and defend it in three sentences.
  6. Take a well-calibrated and a poorly-calibrated model that have the same AUC. Show the Brier score gap. Explain why the downstream consumer notices the difference.

Runnable Example

Longer Connection

Metric choice sits next to:

Metric discipline is not a reporting convention. It is the first place the project either matches the real-world consequence or silently goes adrift.