Multi-Task Learning

One model, several objectives

Multi-task learning (MTL) trains a single model to perform several related tasks simultaneously, rather than training a separate model per task. A classic example: a single network that takes an image and predicts both object category and bounding box location, or a language model fine-tuned to simultaneously predict sentiment, topic, and language of a piece of text.

The tasks are usually different prediction problems — not just different labels on an identical output format (which is the multi-label setting), but genuinely distinct objectives, often with different loss functions, different output types, and sometimes even different data modalities feeding into a shared trunk.

Shared representations

The standard MTL architecture is a shared backbone (also called the trunk or encoder) that processes raw input into a common feature representation, followed by separate task-specific heads — small output layers, one per task — that each take the shared representation and produce that task's prediction. During training, gradients from every task's loss flow back through their own head and then jointly through the shared backbone.

This means the backbone is shaped by signal from every task at once, not just one. If two tasks both benefit from recognizing edges and textures in an image, or both benefit from understanding sentence structure in text, the shared backbone learns a representation useful for both — effectively giving each task access to training signal that originated from the others.

When multi-task learning helps

MTL tends to help when tasks are related enough that they benefit from shared low-level or mid-level features, and when at least one task has limited training data that can benefit from the stronger signal of a data-rich related task. It also helps for deployment efficiency: one shared backbone serving multiple predictions is cheaper to run than N separate models, which matters for latency and memory on resource-constrained systems.

A common real-world pattern is training a shared backbone where one task has abundant labels (say, basic object detection) and another has scarce labels (say, a rare defect classification task) — the data-rich task effectively subsidizes representation learning for the data-poor one.

When multi-task learning hurts: negative transfer

MTL can also actively hurt performance compared to training separate models, a failure called negative transfer. This happens when tasks are unrelated or, worse, actively conflict — the gradient updates that improve one task's loss make the shared representation worse for another. A backbone being pulled in incompatible directions by competing task gradients can end up serving no task as well as a dedicated single-task model would.

Negative transfer is more likely when tasks differ greatly in difficulty or data volume (a huge, easy task can dominate the shared gradient and starve a small, hard one of useful capacity), when task losses are on very different scales (so one task's gradient magnitude swamps the others unless rebalanced), or when the tasks simply don't share underlying structure despite superficially looking related.

Task weighting and balancing gradients

Because tasks can have wildly different loss scales and difficulties, naively summing losses (as in the code example above) often lets one task dominate training. Practical MTL systems weight each task's loss — manually, based on validation performance, or adaptively during training. Techniques like uncertainty weighting (learning a per-task weight based on that task's prediction noise) and GradNorm (dynamically rescaling gradients so no task's gradient magnitude dominates the shared backbone) address this directly, rather than leaving the balance to chance.

Conditions favoring multi-task learning vs. separate models

What's next

Multi-task learning shares a representation across tasks that all have labeled training data. A more extreme version of "sharing knowledge across tasks" is making predictions for a task with no labeled examples at all. The next lesson, Zero-Shot & Few-Shot Learning, covers how large pretrained models make that possible.