Multi-Label Classification

One label vs. several labels

In multi-class classification, each example gets exactly one label from a set of mutually exclusive options — an image is a cat, a dog, or a bird, never more than one. In multi-label classification, each example can have any number of labels simultaneously, including zero or all of them. A news article might be tagged as both "politics" and "economy"; a movie might belong to "comedy," "romance," and "drama" at once; a medical scan might show evidence of multiple co-occurring conditions.

This is a fundamentally different prediction target. Instead of predicting one class out of K, the model predicts a subset of a label set of size K — there are 2^K possible subsets, though in practice only a small fraction ever appear in real data.

Binary relevance: the naive baseline

The simplest approach, binary relevance, trains one independent binary classifier per label: a "is this politics?" classifier, a separate "is this economy?" classifier, and so on, each ignoring the others entirely. At inference time, you run every classifier and collect whichever ones fire as positive.

Binary relevance is simple, parallelizable, and often a perfectly adequate baseline. Its weakness is exactly what the name implies: it treats each label as independent, so it cannot use the fact that "politics" and "economy" co-occur far more often than chance, or that some labels almost never appear together. Any correlation structure between labels is thrown away.

Why label correlation matters

Real label sets are rarely independent. In a movie-genre dataset, "action" and "thriller" co-occur constantly, while "children" and "horror" almost never do. A method that models these correlations can use predictions for one label as evidence for or against another, improving accuracy especially on labels with little training data of their own but strong correlation with better-supported labels.

Classifier chains capture this by training label classifiers in sequence, where each classifier receives not just the original features but also the (true, during training; predicted, during inference) values of all previously predicted labels in the chain. This lets later classifiers condition on earlier label decisions, injecting correlation structure that binary relevance ignores — at the cost of being sensitive to chain order and compounding errors if an early prediction is wrong.

Label powerset takes a different approach: it treats each distinct combination of labels observed in training data as its own single class and trains an ordinary multi-class classifier over those combinations. This fully captures label correlation by construction, but the number of distinct combinations can explode and many combinations will have very few training examples, making it impractical for large label sets.

Evaluating multi-label predictions

A single accuracy number doesn't capture multi-label performance well, because partial correctness is common and meaningful — predicting 2 of 3 true labels correctly is clearly better than predicting 0, but exact-match accuracy would score both as failures. Common metrics instead include Hamming loss (the fraction of individual label predictions that are wrong, averaged over all labels and examples), micro-averaged F1 (aggregates true/false positives/negatives across all labels before computing F1, giving every individual label prediction equal weight), and macro-averaged F1 (computes F1 per label, then averages across labels, giving every label equal weight regardless of how often it appears).

Multi-label strategies compared

The neural network approach

Modern multi-label classification with neural networks typically skips these explicit strategies and instead uses a shared feature-extraction backbone followed by K independent sigmoid outputs (rather than one softmax over K classes), trained with binary cross-entropy loss summed or averaged over all labels. The shared backbone lets the network learn features useful across labels, implicitly capturing some correlation structure through the shared representation, without the explicit chaining or combinatorial expansion of classical methods. This is the default pattern in image tagging, document classification, and recommendation-adjacent multi-label problems today.

What's next

Sharing a backbone across multiple output labels is a specific case of a broader idea: training one model to handle several related objectives at once. The next lesson, Multi-Task Learning, generalizes this to tasks that aren't just different labels on the same input but genuinely different prediction problems sharing a representation.