Vision Transformers (ViT)
Treating an image as a sequence of patches
The Vision Transformer (Dosovitskiy et al., 2020) applies the standard transformer encoder architecture to images with minimal modification, by reframing an image as a sequence. An image is split into fixed-size, non-overlapping patches (typically 16x16 pixels), each patch is flattened into a vector and linearly projected into the model's embedding dimension, and the resulting sequence of patch embeddings is fed into a standard transformer encoder exactly as if they were word tokens — plus a learnable [CLS] token prepended to the sequence, whose final representation is used for classification.
Why this breaks from convolutional inductive bias
Convolutional neural networks have strong built-in assumptions — locality (nearby pixels are related) and translation equivariance (a feature detector works the same wherever it appears in the image) — baked directly into the convolution operation. ViT drops these inductive biases almost entirely: self-attention lets every patch attend to every other patch from the very first layer, with no architectural preference for nearby patches over distant ones. This flexibility is a double-edged sword: it can in principle learn whatever relational structure the data actually has, but without the convolutional prior to lean on, ViT needs considerably more training data to learn good visual representations from scratch.
import numpy as np
def patchify(image, patch_size=16):
"""image: (H, W, C) array. Returns (num_patches, patch_size*patch_size*C)."""
H, W, C = image.shape
patches = []
for i in range(0, H, patch_size):
for j in range(0, W, patch_size):
patch = image[i:i+patch_size, j:j+patch_size, :]
patches.append(patch.flatten())
return np.stack(patches)
image = np.random.rand(224, 224, 3)
patches = patchify(image, patch_size=16)
print(f"num patches: {patches.shape[0]}, flattened patch dim: {patches.shape[1]}")
# 224/16 = 14, so 14*14 = 196 patches, each of dim 16*16*3 = 768 before projectionViT needs scale to beat CNNs, not clever tricks
CNN vs Vision Transformer
| Property | CNN | Vision Transformer |
|---|---|---|
| Inductive bias | Strong (locality, translation equivariance) | Minimal, learned from data |
| Data efficiency (small datasets) | Better | Weaker without pretraining or augmentation |
| Scaling with data/compute | Diminishing returns sooner | Scales favorably with more data |
| Receptive field | Grows gradually with depth | Global from the first layer |
| Position information | Implicit via convolution structure | Must be added explicitly (positional embeddings) |
Positional embeddings for patches
Just as transformers for text need positional encoding because self-attention is permutation-invariant, ViT adds a learned positional embedding to each patch embedding before the first transformer layer, so the model knows where each patch sits in the original image grid. Interestingly, ViT performs reasonably well even with fairly simple 1D learned positional embeddings (treating the 2D grid as a flattened sequence) rather than more sophisticated 2D-aware schemes — the model appears able to learn the necessary 2D spatial relationships from data given enough training.
Fixed patch size limits resolution flexibility
What's Next?
Plain ViT's global, uniform attention over all patches is powerful but computationally expensive at high resolution. Architectures that reintroduce some locality and hierarchy into the transformer, while keeping its data-driven flexibility, pick up directly from this tradeoff.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.