Label Smoothing
The problem with hard one-hot targets
In standard multi-class classification, a one-hot target assigns probability 1 to the correct class and 0 to every other class. When you train with cross-entropy loss against these hard targets, the only way to drive loss toward zero is to push the predicted probability for the correct class toward 1 — which, through the softmax function, requires driving the correct class's logit toward $+\infty$ relative to the others. The network is literally rewarded for becoming infinitely confident, even when that confidence isn't warranted. This encourages overfitting and produces poorly calibrated probabilities, where the model's stated confidence doesn't match its actual accuracy.
What label smoothing changes
Label smoothing (introduced for image classification in the Inception-v2/v3 line of work, and widely adopted since) replaces the hard one-hot target with a softened distribution. For a correct class $k$ out of $K$ total classes and smoothing parameter $\varepsilon$, the target becomes: $y_k = 1 - \varepsilon$ for the correct class, and $y_i = \frac{\varepsilon}{K-1}$ for every incorrect class $i$. Instead of training the model to predict probability exactly 1 for the right answer, you train it to predict something like 0.9 for the right answer and spread the remaining 0.1 uniformly across the wrong answers. The loss is now minimized at a finite logit gap rather than at infinity.
import torch
import torch.nn.functional as F
def label_smoothed_cross_entropy(logits, targets, num_classes, epsilon=0.1):
"""
logits: (batch, num_classes) raw model outputs
targets: (batch,) integer class indices
"""
log_probs = F.log_softmax(logits, dim=-1)
# Build the smoothed target distribution
smooth_val = epsilon / (num_classes - 1)
true_dist = torch.full_like(log_probs, smooth_val)
true_dist.scatter_(1, targets.unsqueeze(1), 1.0 - epsilon)
# Cross entropy against a soft distribution: -sum(target * log_prob)
loss = -(true_dist * log_probs).sum(dim=-1)
return loss.mean()
# PyTorch has this built in since 1.10:
# loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)Why this helps generalization and calibration
Label smoothing acts as a regularizer on the logit space: it caps the gap the model is incentivized to create between the correct class's logit and the rest, which keeps weight magnitudes and logit magnitudes from growing unboundedly during training. Empirically, this tends to improve generalization on held-out data for image classification, machine translation, and speech recognition tasks (notably in the original Transformer paper, which uses $\varepsilon = 0.1$ by default). It also produces better-calibrated probability estimates — predicted confidences that more closely track empirical accuracy — because the model is never trained to be literally 100% certain.
A side effect: it can hurt knowledge distillation
Picking epsilon and when to skip it
$\varepsilon = 0.1$ is a common default and rarely needs aggressive tuning — values much above 0.2 start noticeably hurting accuracy on tasks with many fine-grained classes, since the smoothing budget gets spread thin or thick depending on $K$. Label smoothing is most useful when overfitting or overconfidence is a real concern: smaller datasets, many output classes, or tasks downstream of the probabilities (e.g. active learning, uncertainty-based filtering) that need calibrated confidence. It's less critical, and sometimes skipped, in very large-scale pretraining regimes where the sheer volume of data already limits overfitting.
| Technique | Targets loss function | Primary benefit |
|---|---|---|
| Hard one-hot labels | Standard cross-entropy | Simple, but encourages overconfidence |
| Label smoothing | Cross-entropy vs. soft target | Regularization + calibration |
| Knowledge distillation | Cross-entropy vs. teacher's soft outputs | Transfers teacher's learned similarity structure |
| Mixup / CutMix | Cross-entropy vs. blended labels | Regularization via blended inputs, not just targets |
What's Next?
Label smoothing is one way to reshape what cross-entropy loss actually optimizes for; other loss-function modifications, like focal loss for class imbalance or contrastive losses for representation learning, tackle different mismatches between the standard loss and what you actually want the model to learn.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.