Optical Flow Estimation
What optical flow measures
Optical flow is the apparent motion of pixel brightness patterns between two consecutive video frames — for every pixel in frame $t$, a 2D vector $(u, v)$ describing where that pixel's content moved to in frame $t+1$. It underlies video compression, action recognition, video frame interpolation, autonomous driving perception, and stabilization.
The word 'apparent' matters: optical flow measures how brightness patterns move, which is a proxy for true object motion but not identical to it — a rotating, uniformly-lit sphere has zero optical flow despite genuinely moving, and a stationary scene under moving lighting can show nonzero flow.
The brightness constancy assumption
Classical optical flow estimation starts from the brightness constancy assumption: a pixel's intensity doesn't change as it moves between frames, only its position does. Formally, if $I(x, y, t)$ is the image intensity at position $(x,y)$ and time $t$, and the pixel moves by $(dx, dy)$ over time $dt$:
$$I(x, y, t) = I(x+dx, y+dy, t+dt)$$
Taking a first-order Taylor expansion of the right side and simplifying gives the optical flow constraint equation:
$$I_x u + I_y v + I_t = 0$$
where $I_x, I_y$ are the image's spatial gradients, $I_t$ is its gradient over time, and $(u, v)$ is the flow we want to solve for. This is one equation with two unknowns per pixel — underdetermined, which is why it's called the aperture problem: a small local window can tell you motion along the gradient direction but not perpendicular to it.
Solving the aperture problem: more assumptions, more equations
import numpy as np
def lucas_kanade_flow(Ix, Iy, It, window=5):
"""Ix, Iy, It: spatial/temporal gradients at one pixel's neighborhood.
Solves the overdetermined system A @ [u, v] = -b via least squares."""
A = np.stack([Ix.flatten(), Iy.flatten()], axis=1) # shape (window*window, 2)
b = -It.flatten()
# Normal equations: (A^T A) [u, v] = A^T b
ATA = A.T @ A
ATb = A.T @ b
flow = np.linalg.solve(ATA, ATb) # [u, v]
return flow
# A^T A must be well-conditioned (non-degenerate gradients in two directions)
# -- this is exactly why Lucas-Kanade fails on flat, textureless regions.Learned optical flow: FlowNet and RAFT
Modern approaches replace the hand-derived constraint equation with a neural network trained end-to-end on synthetic data with known ground-truth flow. FlowNet was the first end-to-end CNN approach, directly regressing a flow field from a pair of stacked input frames. RAFT (Recurrent All-Pairs Field Transforms, 2020) became the dominant architecture: it builds a 4D correlation volume between every pixel in frame 1 and every pixel in frame 2, then uses a recurrent update operator to iteratively refine the flow estimate, looking up relevant correlations at each step. This all-pairs approach handles large displacements far better than classical local-window methods, which fundamentally can't see motion larger than their window size.
Classical vs learned optical flow
| Property | Lucas-Kanade / Horn-Schunck | RAFT / learned |
|---|---|---|
| Large displacements | Struggles — local window limits range | Handles well via all-pairs correlation |
| Training data needed | None — purely analytical | Large synthetic datasets with ground-truth flow |
| Textureless regions | Fails (degenerate gradients) | Learned priors fill in plausible flow |
| Speed | Very fast, lightweight | Slower, GPU-dependent, but far more accurate |
Occlusion is the hard remaining problem
What's Next?
Optical flow is a specific instance of a more general problem — establishing dense correspondence between two signals — that recurs in stereo depth estimation, video frame interpolation, and 3D scene reconstruction. The same correlation-volume idea behind RAFT shows up, in modified form, wherever a model has to match structure across two related inputs.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.