CLIP and Vision-Language Pretraining
Aligning images and text in one shared space
CLIP (Contrastive Language-Image Pretraining, Radford et al., 2021) trains an image encoder and a text encoder jointly, using a contrastive objective so that an image and its correct caption end up close together in a shared embedding space, while mismatched image-caption pairs end up far apart. It was trained on roughly 400 million image-text pairs scraped from the web — no manual class labels, just naturally occurring captions — which let it learn a remarkably general notion of visual-semantic similarity rather than being tied to a fixed label taxonomy like ImageNet's 1000 classes.
The training objective: a symmetric contrastive loss
For a batch of $N$ image-text pairs, CLIP computes all $N \times N$ possible image-text similarity scores (cosine similarity between the image encoder's output and the text encoder's output, scaled by a learned temperature), then applies a contrastive loss in both directions: for each image, the correct caption should score highest among all $N$ captions in the batch; for each caption, the correct image should score highest among all $N$ images. The final loss is the average of the image-to-text and text-to-image cross-entropy losses, both computed over the same similarity matrix.
import numpy as np
def clip_loss(image_embeds, text_embeds, temperature=0.07):
"""image_embeds, text_embeds: (N, d) arrays, L2-normalized rows."""
logits = (image_embeds @ text_embeds.T) / temperature # (N, N) similarity matrix
N = logits.shape[0]
labels = np.arange(N) # correct pairing is the diagonal
def cross_entropy_rows(logit_matrix):
exp = np.exp(logit_matrix - logit_matrix.max(axis=1, keepdims=True))
probs = exp / exp.sum(axis=1, keepdims=True)
return -np.mean(np.log(probs[labels, labels] + 1e-9))
loss_i2t = cross_entropy_rows(logits) # each image finds its text
loss_t2i = cross_entropy_rows(logits.T) # each text finds its image
return (loss_i2t + loss_t2i) / 2
# L2-normalize embeddings before computing similarity, as CLIP does
embeds_img = np.random.randn(4, 8); embeds_img /= np.linalg.norm(embeds_img, axis=1, keepdims=True)
embeds_txt = np.random.randn(4, 8); embeds_txt /= np.linalg.norm(embeds_txt, axis=1, keepdims=True)
print(f"CLIP loss: {clip_loss(embeds_img, embeds_txt):.4f}")Zero-shot classification: the headline capability
Because CLIP learns a shared space rather than a fixed classifier head, it can perform zero-shot image classification on entirely new label sets without any task-specific training: embed the image, embed candidate text prompts like 'a photo of a {class_name}' for every class of interest, and pick the class whose text embedding has the highest cosine similarity to the image embedding. This flexibility — swap in any class names at inference time, no retraining needed — was a genuine departure from the fixed-taxonomy paradigm that dominated computer vision before it, and it generalized surprisingly well to distribution shifts that typically break conventionally trained classifiers.
Prompt engineering transfers to vision through CLIP
CLIP vs traditional supervised image classifiers
| Property | Traditional classifier (e.g. ResNet on ImageNet) | CLIP |
|---|---|---|
| Label format | Fixed integer class IDs | Free-text descriptions |
| New class support | Requires retraining or fine-tuning the head | Zero-shot, just write a new prompt |
| Training data | Manually labeled dataset | Naturally occurring web image-text pairs |
| Robustness to distribution shift | Often weak | Notably stronger out-of-the-box |
CLIP inherits the biases of web-scraped data
What's Next?
CLIP's text-image alignment became foundational infrastructure for generative vision-language systems — it's the backbone that lets text-to-image diffusion models understand what a text prompt is asking for in visual terms, and understanding the vision side of that pipeline starts with how transformers process images in the first place.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.