Confusion Matrix Deep Dive
The one table every classification metric is built from
A confusion matrix tabulates predicted class against true class, and nearly every classification metric you'll ever use — accuracy, precision, recall, F1, specificity — is just a different ratio computed from its cells. For binary classification the four cells are: True Positives (TP) (predicted positive, actually positive), False Positives (FP) (predicted positive, actually negative — a 'false alarm'), False Negatives (FN) (predicted negative, actually positive — a 'miss'), and True Negatives (TN) (predicted negative, actually negative). Understanding these four cells directly, rather than just memorizing formulas, is what lets you reason about *which* metric actually matters for a given business problem.
Deriving precision, recall, and F1 from first principles
Precision $= \frac{TP}{TP + FP}$ asks: of everything the model flagged as positive, how much was actually positive? It's the metric to optimize when false positives are costly (e.g., flagging a legitimate transaction as fraud and blocking a real customer). Recall (sensitivity) $= \frac{TP}{TP + FN}$ asks: of everything that was actually positive, how much did the model catch? It's the metric to optimize when false negatives are costly (e.g., missing an actual tumor in a screening test). F1 $= 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$, the harmonic mean, penalizes a model that's great at one but terrible at the other far more than a simple average would.
from sklearn.metrics import confusion_matrix, classification_report
import numpy as np
y_true = np.array([1,0,1,1,0,1,0,0,1,0,1,1,0,0,1])
y_pred = np.array([1,0,0,1,0,1,1,0,1,0,1,0,0,1,1])
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
specificity = tn / (tn + fp)
print(f"Precision={precision:.3f} Recall={recall:.3f} F1={f1:.3f} Specificity={specificity:.3f}")
print(classification_report(y_true, y_pred)) # sanity check against sklearn's own computation
Accuracy is dangerously misleading on imbalanced data
Extending to multiclass
For $K$ classes, the confusion matrix becomes a $K \times K$ grid, with the diagonal showing correct predictions and off-diagonal cells showing exactly which classes get confused with which — this is often more diagnostically useful than any single aggregate metric, since it shows the *direction* of confusion (e.g., 'cat' is frequently mistaken for 'dog' but never for 'car'). Per-class precision/recall can be averaged either as a macro average (unweighted mean across classes, treating rare classes equally) or weighted/micro average (weighted by class frequency) — the choice materially changes the reported number on imbalanced multiclass problems and should be chosen deliberately, not left at a library's default.
| Metric | Formula | Optimize when... |
|---|---|---|
| Precision | TP / (TP + FP) | False positives are costly |
| Recall | TP / (TP + FN) | False negatives are costly |
| F1 | 2·P·R / (P+R) | Both matter, need single balanced number |
| Specificity | TN / (TN + FP) | Correctly ruling out negatives matters |
What's next?
The confusion matrix is computed at one fixed decision threshold, but most classifiers output a continuous score before that threshold is applied. Sweeping that threshold across its full range — and plotting how precision, recall, and the false-positive rate trade off as it moves — is the natural next tool for evaluating a classifier's behavior independent of any single threshold choice.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.