Self-Supervised Learning

The labeled data bottleneck

Supervised learning needs labeled examples, and labeled examples are expensive: someone has to annotate each one. This put a hard ceiling on how much data a model could realistically train on, no matter how much unlabeled data existed. Self-supervised learning (SSL) sidesteps this entirely by generating the labels automatically from the structure of the unlabeled data itself. The model is given a task where the 'correct answer' can be derived mechanically from the raw data — no human annotation needed — yet solving that task forces the model to learn genuinely useful representations of the input.

Pretext tasks: manufacturing supervision from nothing

A pretext task is a self-generated prediction problem constructed from unlabeled data, designed purely to force a model to learn useful structure — nobody actually cares about the pretext task's output itself. You've already seen two major examples:

- Masked language modeling (BERT): hide some tokens, predict them from context. The 'label' for each masked position is just the original token, already present in the raw text.
- Next-token prediction (GPT): predict the next word given everything before it. The label is literally the next word in the raw text.

In computer vision, classic pretext tasks included predicting the rotation applied to an image, filling in a masked patch, or determining whether two image patches came from the same original image. In every case, the label comes for free from the data itself — the only 'annotation' is a mechanical transformation, not a human judgment.

Contrastive learning: SimCLR's approach

A particularly influential family of self-supervised methods in vision is contrastive learning. SimCLR (2020) illustrates the core idea cleanly: take an unlabeled image, apply two different random augmentations to it (crop, color jitter, blur) to create two different 'views' of the same underlying image, and pass both through the same encoder network. Since both views came from the same original image, their representations should be similar. Representations of views coming from *different* images in the batch should be dissimilar.

The training objective — a contrastive loss — directly encodes this: pull together the representations of augmented pairs from the same source image ('positive pairs'), and push apart representations of pairs from different source images ('negative pairs'). No human ever labeled what's in any image; the only supervision is the mechanical fact that two augmented crops came from the same original photo.

Why this is exactly how modern LLMs are trained too

It's worth stating plainly: next-token prediction, the pretraining objective behind every GPT-style LLM, is itself a self-supervised pretext task. The 'label' for predicting the next word is simply the next word that already exists in the training text — no human annotated it. This is precisely why LLM pretraining can scale to trillions of tokens scraped from the internet: the supervision signal is free, derived mechanically from text that already exists, rather than something that has to be manually created at the same scale.

This is the throughline connecting ELMo/BERT-style masked prediction, GPT-style next-token prediction, and SimCLR-style contrastive image learning: all three are self-supervised, all three manufacture a learning signal from unlabeled data's own structure, and all three produce representations that transfer broadly to downstream tasks.

Self-supervised pretext tasks across modalities

What's next

Self-supervised pretraining produces a general-purpose model, but that model still needs to be adapted to the task you actually care about. That adaptation process -- and the broader principle of reusing a pretrained model rather than training from scratch -- is the subject of the next lesson, transfer learning.