Neural Radiance Fields (NeRF)

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

A radically different kind of 3D representation

Traditional 3D representations — meshes, voxel grids, point clouds — explicitly store geometry: a list of vertices, a grid of occupied cells, a cloud of 3D points. Neural Radiance Fields (NeRF, Mildenhall et al., 2020) represent a scene completely differently: as the weights of a small neural network that implicitly encodes the scene, with no explicit geometry stored anywhere. Given a 3D point and a viewing direction, the network outputs that point's color and density — and the entire visible scene, from any viewpoint, can be reconstructed by querying this function repeatedly along camera rays.

The function NeRF learns

A NeRF is a multilayer perceptron $F_\theta$ that maps a 5D input — a 3D position $(x, y, z)$ and a 2D viewing direction $(\theta, \phi)$ — to a 4D output: an RGB color $(r, g, b)$ and a scalar volume density $\sigma$:

$$F_\theta(x, y, z, \theta, \phi) \rightarrow (r, g, b, \sigma)$$

The density $\sigma$ represents how opaque that point in space is — near zero in empty air, high at an actual surface. Crucially, the output depends on viewing direction as well as position, which lets the network represent view-dependent effects like specular highlights and reflections that genuinely look different depending on where you're looking from.

Rendering: integrating color along a ray

To render a single pixel, NeRF casts a ray from the camera through that pixel into the scene, samples many points along the ray, queries the network at each sampled point for its color and density, and combines them using classical volume rendering: each point's contribution to the final pixel color is weighted by its own density and by how much of the ray's energy has survived (not been absorbed) by earlier, closer points along the ray. This is numerically approximated as a weighted sum over discrete samples — points behind an opaque surface contribute almost nothing, because earlier high-density samples have already 'blocked' the ray.

volume_rendering.py

python

import numpy as np

def render_ray(colors, densities, delta_t):
    """colors: (N, 3) RGB at each sampled point along the ray.
    densities: (N,) volume density (sigma) at each point.
    delta_t: distance between consecutive samples.
    Returns the rendered pixel color via discretized volume rendering."""
    alpha = 1 - np.exp(-densities * delta_t)          # opacity contributed by each sample
    transmittance = np.cumprod(np.concatenate([[1.0], 1 - alpha[:-1]]))  # light surviving to this point
    weights = transmittance * alpha                    # each sample's contribution weight
    pixel_color = np.sum(weights[:, None] * colors, axis=0)
    return pixel_color, weights

Training needs no 3D ground truth at all

NeRF is trained purely from a set of 2D photographs of a scene from known camera poses — no depth sensors, no 3D scans, no manual labeling. For every pixel in every training photo, the network renders its predicted color via the volume rendering equation and backpropagates the squared error against the actual photographed pixel color. Because rendering is fully differentiable end-to-end, gradients flow straight back into the MLP's weights, and the network gradually discovers a 3D density/color field consistent with every photo simultaneously — effectively solving an inverse-rendering problem using only gradient descent.

Explicit mesh vs implicit NeRF representation

Property	Mesh / point cloud	NeRF
Storage	Explicit vertices/points/faces	Implicit — a function (network weights)
View-dependent effects	Requires separate material/shading models	Naturally captured via direction input
Rendering speed	Fast (rasterization, GPU-native)	Slow — requires many network queries per pixel
Editability	Directly editable geometry	Hard to edit — geometry is entangled in weights

The original NeRF's biggest practical problem: speed

Rendering a single frame requires querying the MLP at hundreds of sample points per ray, for every pixel in the image — the original NeRF took tens of seconds per frame and hours to train on a single scene. Follow-up work (Instant-NGP with hash-grid feature encodings, Plenoxels with explicit voxel grids instead of a pure MLP) brought training down to seconds and rendering to real-time by trading some of the 'purely implicit' elegance for explicit cached structures.

What's Next?

NeRF proved that differentiable volume rendering could reconstruct photorealistic 3D scenes from photos alone — but its per-pixel MLP-querying cost became the motivation for an entirely different, explicit representation built for speed: 3D Gaussian splatting.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →