Transfer Learning

Why train from scratch when you don't have to

Training a large neural network from scratch requires enormous amounts of labeled data and compute — resources most practitioners and most tasks simply don't have. Transfer learning is the practice of taking a model already trained on one task (usually a large, general one) and reusing it, with little or no additional training, as the starting point for a different, usually smaller and more specific task.

The underlying assumption is that the earlier layers of a network trained on a large, diverse dataset learn generally useful features — edges and textures in vision, grammar and word relationships in language — that are relevant far beyond the exact task the model was originally trained on. Only the later, more task-specific layers need serious adaptation.

Two main strategies: feature extraction and fine-tuning

Feature extraction freezes the pretrained model entirely and uses it purely as a fixed function that converts raw input into a useful representation (an embedding). You then train a small new model — often just a single linear layer — on top of those frozen features for your specific task. This is cheap, fast, and requires very little labeled data, but it caps performance at whatever the frozen representation already captures; if your task needs the model to notice something the original representation didn't emphasize, feature extraction can't fix that.

Fine-tuning instead continues training the pretrained model's own weights (all of them, or a subset) on the new task's data, usually with a small learning rate to avoid destroying the useful knowledge already encoded. This adapts the representation itself to the new task, typically yielding better performance than feature extraction, but it needs more labeled data and compute, and carries a real risk of catastrophic forgetting — overwriting the general knowledge the pretraining stage captured if the fine-tuning learning rate is too high or the new dataset is too small and narrow.

Feature extraction vs fine-tuning

Why this changed the practitioner's workflow

Before transfer learning was standard practice, building a model for a new task typically meant training from scratch — which in turn meant you needed a large labeled dataset for that specific task before you could get anywhere. Transfer learning inverted the economics: a large, well-resourced lab (or company) pretrains a powerful general-purpose model once, and anyone else can adapt it to their own narrower task with a small fraction of the data and compute that training from scratch would require.

This is the pattern behind essentially the entire modern ML ecosystem: pretrained vision backbones fine-tuned for medical imaging or defect detection, pretrained language models fine-tuned for customer-support chatbots or legal document review, and foundation models adapted via lightweight techniques to countless downstream products. It shifted the practitioner's core skill from 'can I collect enough data to train this from scratch' to 'which pretrained model is the right starting point, and how should I adapt it.'

What's next

Transfer learning tells you how to reuse a pretrained model, but says nothing about how to keep training stable once you're updating weights at scale. The next lesson looks at a technique foundational to training deep networks reliably in the first place -- batch normalization and layer normalization.