Contrastive Learning
Learning representations by comparison, not labels
Contrastive learning trains a model to produce embeddings where similar ('positive') examples end up close together in representation space, and dissimilar ('negative') examples end up far apart — without needing explicit class labels. The supervisory signal comes entirely from knowing which pairs should be similar, which is often constructible for free from the data's own structure: two augmented crops of the same image, two sentences that are paraphrases, or a user and the item they clicked. This makes contrastive learning a cornerstone of modern self-supervised representation learning.
The InfoNCE loss
The dominant contrastive objective is InfoNCE (Noise Contrastive Estimation): for an anchor example with embedding $q$, one positive embedding $k^+$, and a batch of negative embeddings $k^-_1, ..., k^-_n$, the loss is $L = -\log \frac{\exp(q \cdot k^+ / \tau)}{\exp(q \cdot k^+ / \tau) + \sum_i \exp(q \cdot k^-_i / \tau)}$, where $\tau$ is a temperature parameter controlling how sharply the model is penalized for ranking negatives close to the anchor. This is structurally a softmax classification problem — 'identify the true positive among all these candidates' — which is why contrastive learning scales so well with larger batches: more negatives per batch means a harder, more informative classification task.
import numpy as np
def info_nce_loss(query, positive, negatives, temperature=0.07):
"""query, positive: 1D vectors. negatives: list of 1D vectors."""
def cos_sim(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
pos_score = np.exp(cos_sim(query, positive) / temperature)
neg_scores = sum(np.exp(cos_sim(query, neg) / temperature) for neg in negatives)
return -np.log(pos_score / (pos_score + neg_scores))
query = np.array([0.6, 0.2, 0.1])
positive = np.array([0.55, 0.25, 0.15]) # augmented view of the same instance
negatives = [np.array([-0.3, 0.8, 0.1]), np.array([0.1, -0.5, 0.9])]
loss = info_nce_loss(query, positive, negatives)
print(f"InfoNCE loss: {loss:.4f}")SimCLR and the role of augmentation
SimCLR (Chen et al., 2020) popularized a simple, effective contrastive recipe for images: take a batch of images, create two randomly augmented views of each (random crop, color jitter, blur, flip), and treat the two views of the same image as a positive pair, with every other image in the batch (and its augmented view) serving as negatives. The specific choice and strength of augmentations turned out to matter enormously — augmentations too weak let the model cheat with trivial low-level cues (matching exact pixel colors rather than learning semantic content), while augmentations too strong destroy the semantic signal the model is supposed to preserve.
Negative sampling quality drives representation quality
Contrastive learning approaches
| Method | Negative source | Key requirement |
|---|---|---|
| SimCLR | Other images in the same large batch | Large batch size |
| MoCo | Momentum-updated memory queue | Momentum encoder, decoupled from batch size |
| CLIP | Other image-text pairs in batch | Large, diverse paired dataset |
| SimSiam / BYOL | None (no explicit negatives) | Asymmetric architecture to avoid representation collapse |
Representation collapse without negatives or asymmetry
What's Next?
Contrastive learning works within a single modality, but the same pull-together/push-apart idea generalizes naturally to aligning two entirely different modalities — like images and the text that describes them — in a shared embedding space.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.