A neural network diagram with circles and arrows is a useful abstraction, but it hides what the hardware actually does: almost every layer in a modern network reduces to a matrix multiplication (plus a cheap elementwise operation like adding a bias or applying an activation function). A fully connected layer is literally a matrix multiply between the input vector and a weight matrix. A convolution is a matrix multiply in disguise, restructured to reuse the same small filter across many spatial positions. Attention in a transformer is a sequence of matrix multiplies between queries, keys, and values. If you can estimate the size of the matrix multiplies, you can estimate almost everything about a model's compute cost.

FLOPs: the currency of neural network compute

A FLOP (floating-point operation) is one multiply or one add. Multiplying two matrices of shape (m x k) and (k x n) takes roughly 2 x m x k x n FLOPs — the factor of 2 because each output element requires k multiplications and k-1 additions, which rounds to roughly k multiply-adds, each counted as 2 FLOPs. This simple formula is the building block for estimating the cost of an entire network: sum the FLOPs of each layer's matrix multiply, and you have a reasonable estimate of total compute per forward pass.

python

Training costs roughly 3x a forward pass, per example

A forward pass computes predictions; training also needs a backward pass to compute gradients, which involves roughly two more matrix multiplies of similar size (one for the gradient with respect to the input, one for the gradient with respect to the weights). A commonly used rule of thumb is that one training step costs about 3x the FLOPs of a single forward pass, per example processed. This rule of thumb is the basis for widely used compute-estimation formulas like 'training FLOPs is approximately 6 times the parameter count times the number of training tokens' for transformer-style models — a formula worth knowing by name even without memorizing its derivation, because it's how teams estimate training cost and training time before committing hardware budget.

Note

It's common shorthand to describe a model by parameter count alone ('a 7-billion-parameter model'), but two models with the same parameter count can have very different FLOP costs depending on architecture — a model that reuses parameters across many positions (like a CNN's shared filters) does less compute per parameter than one that applies every parameter once per token (like a dense transformer layer). Parameter count is a decent rough proxy for compute cost within a single architecture family, but it stops being reliable the moment you're comparing across architecture types.

Building compute-budget intuition for model sizing

The practical reason to care about FLOPs isn't academic — it's so you can answer questions like 'can this model hit our latency budget on this hardware' before you've trained it. If a GPU can sustain, say, 100 teraFLOPs/sec of real (not peak theoretical) throughput, and a forward pass for one request costs 2 billion FLOPs, that's a rough ceiling of 50,000 requests/sec from raw compute alone — a number worth sanity-checking against your actual latency and throughput requirements long before investing in training a model that can't be served within budget. This kind of back-of-envelope FLOP accounting is a standard early step in any serious model-sizing exercise.

text

Peak vs. sustained hardware throughput

Concept	What it means	Why it matters for sizing
Peak FLOPs/sec	Theoretical max from hardware spec sheet	Almost never achieved in practice
Sustained FLOPs/sec	Real achievable throughput on typical workloads	The number to actually budget against
Utilization ratio	Sustained ÷ peak, often 20-60% for real workloads	A low ratio signals a non-compute bottleneck (see System Balance)

What's next

FLOP accounting tells you how expensive a given architecture is, but it doesn't tell you which architecture to pick in the first place. The next lesson, Network Architecture Design Tradeoffs, looks at the depth-vs-width and compute-vs-accuracy decisions that shape a model before a single FLOP is spent training it.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →