Neural Architecture Search (NAS)
Automating architecture design
Deciding how many layers a network should have, what kernel sizes to use, how wide each layer should be, and how components connect has traditionally been a manual, trial-and-error process driven by researcher intuition and compute-expensive experimentation. Neural Architecture Search (NAS) reframes architecture design itself as an optimization problem: define a search space of possible architectures, define an objective (usually validation accuracy under a compute/latency budget), and let an automated search procedure find good architectures within that space rather than relying purely on human design.
The three components of any NAS method
Every NAS approach needs to specify: (1) a search space — what architectural choices are even possible (number of layers, operation types per layer, connectivity patterns); (2) a search strategy — how candidate architectures are proposed and explored (reinforcement learning, evolutionary algorithms, gradient-based relaxation, random search); and (3) a performance estimation strategy — how a candidate's quality is evaluated, which is the expensive part, since naively training each candidate from scratch to convergence is prohibitively costly when the search explores thousands of candidates.
NAS search strategies compared
| Strategy | How it works | Main cost driver |
|---|---|---|
| Reinforcement learning (NASNet) | Controller RNN samples architectures, reward = validation accuracy | Thousands of full training runs |
| Evolutionary search | Mutate/crossover a population of architectures, select the fittest | Large population × many generations |
| Random search | Sample architectures uniformly from the space | Surprisingly competitive baseline, still costly |
| Gradient-based (DARTS-style) | Relax discrete choices into continuous weights, optimize with gradients | Single training run, but memory-heavy |
The original cost problem
Reducing the cost: weight sharing and proxy tasks
Later NAS methods made search tractable by avoiding full training of every candidate. Weight sharing approaches (ENAS, one-shot NAS) train a single large 'super-network' containing all possible architectural choices as sub-paths, and candidate architectures are evaluated by activating the relevant sub-path within the shared weights rather than training from scratch. Proxy tasks evaluate candidates on smaller datasets, fewer epochs, or smaller resolutions as a cheap approximation of final performance, trading some ranking accuracy for massive speedups. Performance predictors — small models trained to predict final accuracy from early training curves or architecture encodings — push this further, estimating quality without even running substantial training.
def nas_search(search_space, budget, train_proxy, evaluate):
"""High-level evolutionary NAS loop (conceptual)."""
population = [search_space.sample_random() for _ in range(budget.pop_size)]
for generation in range(budget.generations):
scores = []
for arch in population:
model = train_proxy(arch, epochs=budget.proxy_epochs) # cheap partial training
scores.append(evaluate(model))
# keep the fittest, mutate/crossover to produce the next generation
survivors = select_top_k(population, scores, k=budget.pop_size // 2)
population = survivors + [mutate(a) for a in survivors]
return population[argmax(scores)]What NAS actually found, and its diminishing returns
NAS-discovered architectures like NASNet, AmoebaNet, and EfficientNet (whose scaling rule was itself partly NAS-derived) did outperform hand-designed contemporaries on accuracy-per-FLOP at the time. But the field has also found that random search, given the same evaluation budget, is a surprisingly strong baseline — much of NAS's early apparent advantage came from well-designed search spaces (which still encode substantial human expertise) rather than the search algorithm itself. This has shifted some research emphasis from 'better search strategies' to 'better search spaces and benchmarks for fair comparison.'
What's Next?
Making the discrete architecture search differentiable — so gradients, not sampled candidates, drive the search — is the key idea that made NAS dramatically cheaper, and is worth examining in its own right as a distinct technique.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.