A common misconception is that architecture design is about finding the one network structure that performs best, full stop. In practice, nearly every architecture decision is made under a fixed budget — a latency ceiling, a memory ceiling, a training compute ceiling, or all three — and the real question is which structure gets the most accuracy out of that specific budget. A 2-billion-parameter model and a 20-billion-parameter model aren't competing on 'which is better'; they're answering different questions about how much compute you're willing to spend per prediction.

Depth vs. width: two ways to add capacity

Given a fixed parameter budget, you can add capacity by making a network deeper (more layers) or wider (more units per layer). Depth lets a network build up increasingly abstract representations by composing many simple transformations, and empirically tends to help more with complex, hierarchical problems like image recognition or language understanding. Width increases the capacity of each individual layer, is easier to parallelize (wide matrix multiplies use hardware more efficiently than many sequential narrow ones), and is more resistant to the vanishing-gradient problems that very deep networks without careful design (residual connections, normalization) can suffer from. Most successful modern architectures aren't purely deep or purely wide — they're deep enough to build hierarchical representations and wide enough to keep each layer's matrix multiplies hardware-efficient.

text

Note

For large transformer-based models, researchers found that model performance follows fairly predictable power-law relationships with model size, dataset size, and training compute — meaning you can often predict how much a model will improve from being made bigger, or trained longer, before actually doing it. This is why 'scale up the same basic architecture' became a dominant strategy in large language model development: once you trust the scaling curve, architecture search gives way to compute allocation — deciding how to split a fixed compute budget between model size and training data, rather than inventing new structures.

Compute cost vs. accuracy isn't a straight line

Accuracy gains from added compute follow diminishing returns almost universally: doubling a model's size rarely doubles its accuracy, and at some point additional compute buys vanishingly small improvements. This is why architecture design is fundamentally an optimization problem against a cost curve, not a search for a ceiling-less 'best' — the right stopping point depends on what a marginal percentage point of accuracy is actually worth for the product, weighed against the marginal compute cost (training cost, inference cost, latency) of getting there.

Neural architecture search: automating the tradeoff

Neural Architecture Search (NAS) treats the architecture itself as a thing to optimize, using algorithms (reinforcement learning, evolutionary search, or gradient-based relaxations) to explore combinations of depth, width, layer types, and connectivity patterns, scored against both accuracy and a hardware cost constraint (latency, FLOPs, memory). NAS is expensive — it can cost many times more compute than training a single final model, because it's effectively training or evaluating many candidate architectures — which is why it's typically reserved for cases where the resulting architecture will be deployed at massive scale (mobile phone chips, widely-used vision backbones), amortizing the search cost over millions of inferences.

Manual architecture design vs. neural architecture search

Dimension	Manual design	Neural architecture search
Upfront cost	Low (engineer judgment + a few experiments)	Very high (many candidate models trained/evaluated)
When it pays off	Research, prototyping, most production cases	Massive-scale deployment where search cost amortizes
Optimizes for	Whatever metrics the engineer explicitly checks	Explicit multi-objective score (accuracy + hardware cost)
Reproducibility of result	Depends on documented reasoning	Depends on logged search process and search space

Note

An architecture optimized (manually or via NAS) for low latency on a mobile NPU can perform poorly on a server GPU, and vice versa, because the operations that are cheap on one piece of hardware aren't necessarily cheap on another — depthwise convolutions, for instance, are mobile-friendly but often underutilize a GPU's parallel compute units. Architecture decisions made without a specific deployment target in mind tend to need revisiting once that target is known.

What's next

Once an architecture is chosen, the next scaling challenge is mechanical rather than architectural: how do you actually train a model too large to fit on one GPU? The next lesson, Distributed Training: Data & Model Parallelism, covers how training gets split across many devices.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →