Focal Loss for Class Imbalance
Why plain cross-entropy struggles with imbalance
In a heavily imbalanced dataset — say, 1 positive example for every 1000 negatives, common in object detection and fraud/anomaly tasks — standard cross-entropy loss is dominated by the overwhelming number of easy negatives. Each easy, correctly-classified negative contributes a small but non-zero loss, and summed across thousands of them, that total can swamp the gradient signal coming from the rare, hard positives the model actually needs to learn from. The model can achieve a deceptively low average loss just by getting good at the easy majority class while barely learning the minority class at all.
Focal loss: down-weighting easy examples
Focal loss (Lin et al., 2017, introduced for the RetinaNet object detector) modifies cross-entropy by adding a modulating factor that shrinks the loss contribution from examples the model already classifies confidently and correctly. For binary classification with predicted probability $p_t$ for the true class, standard cross-entropy is $CE = -\log(p_t)$. Focal loss is: $FL(p_t) = -(1-p_t)^\gamma \log(p_t)$. When $p_t$ is close to 1 (an easy, well-classified example), $(1-p_t)^\gamma$ is close to 0, so that example's loss — and more importantly its gradient contribution — is heavily suppressed. When $p_t$ is low (a hard or misclassified example), $(1-p_t)^\gamma \approx 1$ and the loss behaves close to ordinary cross-entropy.
import torch
import torch.nn.functional as F
def focal_loss(logits, targets, alpha=0.25, gamma=2.0, reduction="mean"):
"""
logits: (N,) raw scores for the positive class (binary case)
targets: (N,) binary labels {0, 1}
"""
p = torch.sigmoid(logits)
ce = F.binary_cross_entropy_with_logits(logits, targets.float(), reduction="none")
p_t = p * targets + (1 - p) * (1 - targets) # prob assigned to the true class
modulating_factor = (1 - p_t) ** gamma
alpha_t = alpha * targets + (1 - alpha) * (1 - targets) # class-balance weight
loss = alpha_t * modulating_factor * ce
return loss.mean() if reduction == "mean" else loss.sum()The role of gamma and alpha
$\gamma$ (the focusing parameter) controls how aggressively easy examples get down-weighted — the RetinaNet paper found $\gamma = 2$ worked best across its benchmarks; $\gamma = 0$ recovers ordinary cross-entropy exactly. $\alpha$ is a separate, simpler class-balancing weight (distinct from the focusing mechanism) that scales the loss up for the minority class and down for the majority class, similar to standard class weighting. The two serve different purposes: $\alpha$ addresses the raw frequency imbalance, while $\gamma$ specifically addresses the *difficulty* imbalance — the fact that most negatives aren't just numerous but also trivially easy.
Focal loss reshapes gradients, not just final loss values
| Approach | Mechanism | Typical use case |
|---|---|---|
| Class-weighted cross-entropy | Static weight per class (inverse frequency) | Moderate imbalance, simple fix |
| Focal loss | Dynamic down-weighting of easy examples | Extreme imbalance + many trivial negatives (detection) |
| Oversampling (SMOTE, etc.) | Synthesize/duplicate minority examples | Tabular data, moderate dataset sizes |
| Threshold tuning (post-hoc) | Adjust decision threshold after training | When retraining the loss isn't feasible |
Gamma too high can hurt convergence
What's Next?
Focal loss is one answer to the broader problem of class imbalance; cost-sensitive learning and threshold tuning attack the same problem from the evaluation and decision-boundary side rather than the loss function itself, and the right combination depends on whether you can afford to retrain.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.