Gaussian Splatting for 3D Rendering

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Trading NeRF's implicit function for an explicit, fast one

NeRF represents a scene as a neural network that must be queried hundreds of times per pixel to render a single frame — accurate, but slow. 3D Gaussian Splatting (Kerbl et al., 2023) represents a scene instead as an explicit collection of millions of 3D Gaussian 'blobs,' each with a position, size/orientation, color, and opacity. Rendering becomes a matter of projecting these Gaussians onto the camera's image plane and compositing them — a GPU-friendly operation closer to traditional rasterization than to neural network inference, which is why Gaussian splatting can render in real time at high resolution where NeRF could not.

What a 3D Gaussian actually stores

Each Gaussian primitive carries: a 3D mean position $\mu$ (its center in space), a 3D covariance matrix (defining its size and orientation as an ellipsoid — tight and thin for sharp edges, large and round for soft blurry regions), an opacity value, and a color, often represented with spherical harmonics coefficients so the color can vary by viewing direction (capturing the same view-dependent specular effects NeRF modeled via its direction input). A typical scene reconstruction ends up with anywhere from hundreds of thousands to several million of these Gaussians.

Rendering by splatting and alpha-compositing

To render a frame, every 3D Gaussian is projected ('splatted') onto the 2D image plane given the current camera pose, producing a 2D elliptical footprint for each one. The Gaussians overlapping a given pixel are sorted by depth and alpha-composited front-to-back — exactly like layering semi-transparent stickers, where each successive layer's visible contribution depends on how much opacity the layers in front of it have already consumed. This whole pipeline is implemented as a highly optimized, differentiable tile-based rasterizer, which is the key engineering contribution that makes training fast as well as rendering fast.

alpha_compositing_sketch.py

python

import numpy as np

def composite_pixel(sorted_gaussians_front_to_back):
    """Each entry: (color: (3,), opacity: float) for Gaussians covering this pixel,
    already sorted nearest-camera first."""
    accumulated_color = np.zeros(3)
    transmittance = 1.0  # fraction of light not yet absorbed by nearer Gaussians

    for color, opacity in sorted_gaussians_front_to_back:
        accumulated_color += transmittance * opacity * color
        transmittance *= (1 - opacity)
        if transmittance < 1e-4:
            break  # fully occluded by now, remaining Gaussians contribute nothing

    return accumulated_color

Training is optimization, not a single forward pass

Like NeRF, Gaussian splatting is optimized from a set of calibrated photographs by backpropagating the rendering loss against real pixels — differentiable rasterization makes every Gaussian's position, shape, color, and opacity a trainable parameter. Training also includes an adaptive density control step: Gaussians that are consistently too large to represent fine detail get split into smaller ones, and Gaussians that end up contributing almost nothing (near-zero opacity, or redundant with neighbors) get pruned away — so the final Gaussian count and placement is itself learned, not fixed upfront.

NeRF vs 3D Gaussian Splatting

Property	NeRF	3D Gaussian Splatting
Scene representation	Implicit (MLP weights)	Explicit (millions of Gaussian primitives)
Rendering method	Per-pixel ray marching + MLP queries	Rasterization-style splatting + compositing
Rendering speed	Seconds per frame typically	Real-time (60+ fps on consumer GPUs)
Editability	Hard — geometry entangled in weights	Easier — Gaussians can be directly manipulated

Explicit storage is the tradeoff for speed

A trained Gaussian splatting scene can take hundreds of megabytes to gigabytes to store, since every one of the millions of Gaussians' parameters is stored explicitly — unlike a NeRF's fixed, compact MLP weight file. Active research (compression via vector quantization, pruning more aggressively, hierarchical level-of-detail) targets this storage cost directly, since it's now the main practical bottleneck rather than render speed.

What's Next?

Gaussian splatting's explicit, differentiable, GPU-native representation has made real-time novel-view synthesis practical for the first time, and it's now being extended toward dynamic (moving) scenes and combined with generative models to synthesize entirely new 3D content rather than only reconstructing captured ones.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →