The loss function a neural network minimizes is a function of every one of its parameters — for a model with a billion weights, that's a billion-dimensional surface. We obviously can't visualize a billion-dimensional space directly, but loss landscape visualization techniques project slices of this surface down into 1 or 2 dimensions we can actually plot, to build intuition about why optimization works as well as it does, why some architectures train more easily than others, and how sharp or flat the minima that training finds actually are.

The filter-normalized random direction method

A naive approach — picking two random directions in parameter space and plotting the loss as you move along them — produces misleading plots, because different layers' weights have wildly different natural scales, and a 'random direction' dominated by one layer's scale doesn't meaningfully represent movement in the others. The widely used fix, introduced by Li et al. (2018), is filter-normalized random directions: each random direction vector is rescaled filter-by-filter (or layer-by-layer) to match the norm of the corresponding weights in the actual trained model, before being used as an axis for the plot. This produces loss surface visualizations that are comparable across different architectures and training setups in a way naive random directions are not.

loss_surface_slice.py

python

import torch, copy
import numpy as np

def filter_normalized_direction(model):
    direction = {}
    for name, param in model.named_parameters():
        d = torch.randn_like(param)
        # Rescale so each filter's direction norm matches that filter's weight norm
        if param.dim() > 1:
            d = d * (param.norm(dim=tuple(range(1, param.dim())), keepdim=True) /
                      (d.norm(dim=tuple(range(1, param.dim())), keepdim=True) + 1e-10))
        direction[name] = d
    return direction

def loss_surface_2d(model, loss_fn, data, alphas, betas):
    theta0 = copy.deepcopy(model.state_dict())
    dir1, dir2 = filter_normalized_direction(model), filter_normalized_direction(model)
    surface = np.zeros((len(alphas), len(betas)))

    for i, a in enumerate(alphas):
        for j, b in enumerate(betas):
            perturbed = {k: theta0[k] + a * dir1[k] + b * dir2[k] for k in theta0}
            model.load_state_dict(perturbed)
            surface[i, j] = loss_fn(model, data).item()

    model.load_state_dict(theta0)   # restore original weights
    return surface

Sharp vs. flat minima

One of the most consequential findings from loss landscape visualization is that not all minima the loss reaches zero (or near-zero training loss) at are equally good. Sharp minima — where loss rises steeply in nearby directions — tend to generalize worse to unseen data than flat minima, where loss stays low across a broader neighborhood of parameter space. The intuition: a flat minimum is more robust to the small mismatches between training and test data distributions, since nearby points in parameter space (which the test distribution effectively 'probes' slightly differently than training data would) still have low loss. This observation motivated optimization techniques like Sharpness-Aware Minimization (SAM), which explicitly biases training toward flatter regions of the loss surface rather than just the nearest minimum.

What architectural/training choices tend to do to the loss landscape

Factor	Typical effect on landscape	Observed consequence
Residual (skip) connections	Smooths the landscape, reduces chaotic structure	Easier optimization in very deep networks
Batch/layer normalization	Reduces sharpness, improves conditioning	More stable training, higher learning rates tolerated
Larger batch size (without adjustment)	Tends to find sharper minima	Can generalize worse unless learning rate/regularization is adjusted
SGD with momentum vs. plain SGD	Smoother effective trajectory across landscape	Faster, more stable convergence in practice

2D slices are illustrative, not literal

A 2D loss surface plot, however useful for intuition, is still just two directions sliced out of a billion-dimensional space — it can't capture the full geometry, and different random direction choices produce visibly different-looking plots for the same model. These visualizations are best treated as qualitative, suggestive evidence (e.g., 'this architecture's landscape looks smoother than that one's') rather than precise quantitative measurements of landscape properties.

Why deep networks aren't hopelessly non-convex in practice

Deep network loss landscapes are provably non-convex and, in theory, riddled with the possibility of bad local minima and saddle points. Empirically, though, well-designed architectures trained with standard methods tend to find solutions that behave much better than worst-case non-convex theory would predict — a gap between theoretical pessimism and practical success that loss landscape visualization helped make concrete and is still not fully explained by theory.

What's next?

Loss landscape geometry connects directly to a puzzling empirical phenomenon: models sometimes get *worse* on test data as they're made larger or trained longer, before getting better again — the double descent phenomenon, which complicates the simple 'more capacity, more overfitting' intuition classical learning theory would suggest.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →