Zero-Shot & Few-Shot Learning

Learning without a labeled training set

Traditional supervised learning assumes you have a reasonably sized labeled dataset for the exact task at hand. Zero-shot learning asks a model to perform a task it was never explicitly trained on, given only a description of the task — no labeled examples at all. Few-shot learning relaxes this slightly: the model is given a handful of labeled examples (often 1 to a few dozen), far too few to train a conventional model from scratch, and must generalize from them.

Both settings would be essentially impossible for a model trained from random initialization on a narrow task — there simply isn't enough signal. What makes them work in practice is that the model isn't starting from scratch: it's a large model pretrained on a vast, diverse corpus, and zero/few-shot performance draws on knowledge already encoded during that pretraining.

How large pretrained models enabled this

Large language models and large vision-language models are pretrained on enormous, broad corpora using self-supervised objectives (like predicting the next token, or matching images to captions) that don't require task-specific labels at all. During this pretraining, the model implicitly absorbs an enormous range of patterns: facts, task formats, linguistic structure, associations between concepts, even what a typical instruction and its expected answer look like.

This pretraining effectively front-loads the "learning" that used to happen per-task. When you then ask the model a new task zero-shot, you're not asking it to learn anything new — you're asking it to recognize which of the many patterns it already absorbed applies here. CLIP, for instance, is pretrained by matching images to their captions across hundreds of millions of pairs; this alone gives it enough aligned visual-textual understanding to classify images into categories it was never explicitly trained to recognize, just by comparing an image to text descriptions of candidate labels.

In-context learning

Large language models support a specific, powerful form of few-shot learning called in-context learning: you place a handful of example input-output pairs directly in the prompt, followed by a new input, and the model infers the task pattern from those examples without any weight updates at all. No gradient descent runs — the "learning" happens purely through the forward pass attending over the examples in context.

This is qualitatively different from fine-tuning. Fine-tuning changes the model's weights using labeled examples as a training set; in-context learning leaves weights frozen and uses the examples purely as part of the input the model conditions its prediction on. It's cheap (no training run needed), instant (works at inference time), and flexible (swap the examples to swap the task) — but it's also less reliable for tasks far outside what the pretraining distribution resembles, and performance can be sensitive to example order and phrasing.

Zero-shot, few-shot, and fine-tuning compared

Limitations worth knowing

Zero-shot and few-shot performance is bounded by what the pretrained model already encodes, and it degrades for tasks that are highly specialized, require precise numeric reasoning, or depend on proprietary/private knowledge absent from the pretraining corpus. Few-shot prompts are also constrained by context length — you can't fit thousands of examples in a prompt the way a fine-tuning dataset could use thousands of examples — and results can be surprisingly sensitive to which examples you pick and the order you place them in.

What's next

Zero-shot and few-shot learning rely on a pretrained model already having absorbed useful structure. A related question is how the *order* in which a model sees training examples affects what it learns and how efficiently. The next lesson, Curriculum Learning, covers why training on easy examples before hard ones can improve convergence.