Classifying an image is a well-understood problem; classifying a *video clip* into an action — 'running,' 'opening a door,' 'waving' — adds a dimension that single-frame classification can't capture: time. A single frame of someone standing with one leg raised could be the middle of a jump, a kick, or just an awkward standing pose; the distinguishing information is in how the pose changes across neighboring frames. Action recognition models are built specifically to exploit motion and temporal ordering, not just spatial appearance.

Naive baseline: average frame predictions

The simplest possible approach runs an ordinary image classifier on individual sampled frames and averages the predictions across the clip. This baseline is surprisingly competitive for actions that are visually distinctive even in a single frame (e.g. 'playing basketball' — a court and ball are visible regardless of motion), but it fails badly on actions defined primarily by motion pattern rather than scene content (e.g. distinguishing 'sitting down' from 'standing up,' which can look nearly identical frame-by-frame).

3D convolutions: extending kernels into time

The most direct extension of CNNs to video is the 3D convolution: instead of a 2D kernel sliding over height and width, a 3D kernel slides over height, width, *and* a short window of consecutive frames simultaneously. This lets early layers directly learn spatiotemporal patterns — like a specific kind of motion blur or limb trajectory — rather than bolting temporal reasoning on afterward. C3D and I3D are well-known architectures built this way. The tradeoff is cost: 3D convolutions have far more parameters and compute than their 2D counterparts, and need substantially more video data to train well.

python

Two-stream networks: splitting appearance from motion

An alternative to learning motion implicitly via 3D convolutions is to compute it explicitly and feed it as a second input. Two-stream networks run two separate CNNs: a spatial stream that processes individual RGB frames (capturing scene and object appearance), and a temporal stream that processes precomputed optical flow fields — dense maps of pixel motion between consecutive frames — capturing movement directly. The two streams' predictions are combined (often by averaging or a learned fusion layer) for the final classification. This approach often matched or beat early 3D convolution models while being cheaper to train, because optical flow gives the motion stream a strong, explicit signal rather than requiring the network to discover motion patterns from raw pixels alone.

text

Note

Two-stream networks hand the model a strong, explicit motion signal (optical flow) computed by a separate, non-learned algorithm. 3D convolutions instead ask the network to discover motion-relevant patterns itself directly from raw frame sequences, with no explicit motion computation at all. This mirrors a broader pattern in computer vision and machine learning generally: handing a model a useful hand-engineered feature can work well with less data, while letting the model learn everything end-to-end tends to scale better once enough data and compute are available — which is exactly what happened as video transformers overtook both older approaches.

Transformer-based video models

Following the success of Vision Transformers (ViT) on images, video transformers (e.g. ViViT) extend the same self-attention mechanism across both space and time. Rather than hand-coding how spatial and temporal information should combine (as 3D convolutions and two-stream networks implicitly do), a video transformer splits a clip into spatiotemporal patches and lets self-attention learn which patches — across any distance in space or time — are relevant to each other. This gives the model more flexibility to capture long-range temporal dependencies (an action that unfolds over many seconds) but typically requires large-scale video datasets and significant compute to train effectively, echoing the data/compute hunger transformers showed in NLP and image classification.

Three approaches to video action recognition

Approach	How it captures motion	Compute cost	Data requirement
3D convolutions (C3D, I3D)	Learned implicitly via spatiotemporal kernels	High — 3D kernels are parameter-heavy	Large video datasets needed
Two-stream networks	Explicit, via precomputed optical flow	Moderate — optical flow computed separately	Can work with less data due to explicit motion signal
Video transformers (ViViT)	Learned via self-attention across space-time patches	Very high — attention scales with sequence length	Very large datasets, often pretrained on images first

What's next

Action recognition extends CNN and attention-based vision techniques into the time dimension, a theme that also appears in pose estimation tracked across frames. Continue to point clouds and 3D vision to see how computer vision techniques adapt to an entirely different kind of spatial data — unordered 3D points rather than a 2D pixel grid.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →