Swin Transformer Architecture
The quadratic cost problem ViT doesn't solve
Plain Vision Transformer applies self-attention globally across all patches, which costs $O(n^2)$ in the number of patches $n$. For classification on modest-resolution images this is manageable, but it becomes prohibitive for dense prediction tasks (object detection, segmentation) that need high-resolution feature maps with many more patches, and it prevents ViT from naturally producing the multi-scale feature hierarchies that CNN-based detectors and segmentation networks rely on. The Swin Transformer (Liu et al., 2021, 'Shifted Window') was designed specifically to fix both problems while keeping the core strengths of self-attention.
Windowed self-attention: trading global for local, then shifting
Instead of computing attention across the entire image, Swin partitions patches into small, non-overlapping local windows (e.g. 7x7 patches) and computes self-attention only within each window, reducing cost from quadratic in total patch count to linear. The obvious downside — no information flows between windows — is solved by alternating two types of layers: regular windowed attention, then a layer where the window grid is shifted by half a window size before attention is computed again. The shift means windows in consecutive layers overlap differently, so after a couple of layers, information has effectively propagated across window boundaries without ever needing full quadratic global attention.
import numpy as np
def window_partition(x, window_size):
"""x: (H, W, C) feature map -> list of (window_size, window_size, C) windows."""
H, W, C = x.shape
windows = []
for i in range(0, H, window_size):
for j in range(0, W, window_size):
windows.append(x[i:i+window_size, j:j+window_size, :])
return windows
def shift_feature_map(x, shift_size):
"""Cyclically shift the feature map before the next windowed attention layer."""
return np.roll(x, shift=(-shift_size, -shift_size), axis=(0, 1))
feature_map = np.random.rand(28, 28, 16)
windows_regular = window_partition(feature_map, window_size=7)
shifted_map = shift_feature_map(feature_map, shift_size=3) # window_size // 2
windows_shifted = window_partition(shifted_map, window_size=7)
print(f"regular windows: {len(windows_regular)}, shifted windows: {len(windows_shifted)}")Patch merging rebuilds the CNN-style hierarchy
ViT vs Swin Transformer
| Property | Plain ViT | Swin Transformer |
|---|---|---|
| Attention scope | Global, every patch attends to every patch | Local windows, alternating shifted windows |
| Compute scaling with resolution | Quadratic | Linear |
| Feature hierarchy | Single resolution throughout | Multi-scale via patch merging |
| Best fit | Classification, where global context dominates | Detection, segmentation, dense prediction |
Relative position bias instead of absolute positional embeddings
Swin also departs from ViT's absolute positional embeddings, instead adding a learned relative position bias directly to the attention scores within each window, based on the relative offset between two patches. Because windows are small and fixed-size, learning a relative bias table indexed by all possible relative offsets within a window is cheap and tends to generalize better than absolute position embeddings, especially given that the same window of relative positions recurs identically across the whole image.
Window size is a real architectural tradeoff, not a hyperparameter afterthought
What's Next?
Swin shows how to make transformer-based vision backbones efficient enough for dense, pixel-level tasks — which sets up the natural next question of how a network actually turns that kind of rich hierarchical feature map into a precise, per-pixel segmentation output.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.