AI and Games: How AlphaZero Works

From AlphaGo to AlphaZero

In 2016, AlphaGo defeated world champion Lee Sedol at Go, a feat many researchers expected was decades away because Go's branching factor (roughly 250 legal moves per turn, versus about 35 in chess) makes brute-force search hopeless. AlphaGo relied partly on human game records to bootstrap its play. Its successor, AlphaZero, dropped that crutch entirely: it learned Go, chess, and shogi from scratch, given only the rules, by playing millions of games against itself.

Within 24 hours of self-play training, AlphaZero exceeded the strength of the best existing chess engines of the time — engines built on decades of hand-tuned heuristics and opening theory. That result is the headline reason AlphaZero mattered beyond game-playing circles: it showed that a sufficiently general learning algorithm could out-perform decades of accumulated human domain expertise, from a blank slate.

The three ingredients

AlphaZero's architecture combines three distinct ideas, each solving a different piece of the problem. A deep neural network takes a board position and outputs two things: a policy (a probability distribution over which moves look promising) and a value (an estimate of who's winning from this position). Monte Carlo Tree Search (MCTS) uses that network's guesses to intelligently explore a small, promising slice of the astronomically large game tree, rather than exhaustively searching it. Self-play reinforcement learning generates the training data: the network plays against itself, and the outcomes of those games are used to improve the network's future guesses, in a continuous feedback loop with no human games required.

Monte Carlo Tree Search, guided by the network

Classic MCTS builds a search tree by repeatedly running four steps: selection (walk down the tree toward promising, under-explored moves), expansion (add a new node for an unexplored move), simulation (play out randomly to the end from that node, in pure MCTS), and backpropagation (update the win/loss statistics along the path taken). Classic MCTS used truly random simulations to estimate a position's value — noisy, but unbiased over many trials.

AlphaZero's key modification: instead of random rollouts, it uses the neural network's value output directly as the position estimate, and uses the network's policy output to bias which moves get explored first. This makes the search dramatically more sample-efficient — it spends its search budget on moves the network already believes are promising, rather than wasting it on random play.

Self-play as a training data generator

AlphaZero never sees a single human game. Instead, the current best version of the network plays against itself repeatedly, using MCTS-improved move probabilities at each step. At the end of each game, the actual outcome (win, loss, or draw) becomes a training label: the network is updated to make its value predictions match the real outcome, and its policy predictions match the move probabilities MCTS actually settled on during search.

This is a form of reinforcement learning without any external reward shaping or human labels — the only ground truth is the final game result, which is unambiguous in a deterministic, perfect-information game. Self-play also automatically generates a curriculum: as the network improves, its opponent (itself) improves too, so training examples stay challenging at every stage rather than becoming trivially easy or impossibly hard.

AlphaZero vs. traditional game engines

Why it generalized beyond Go

AlphaZero's recipe is domain-agnostic by design: it requires only a simulator (the rules of the game) and a way to tell who won. That's a much weaker requirement than needing hand-crafted heuristics or human game records, which is why the exact same architecture, with no game-specific tuning beyond the input representation, mastered Go, chess, and shogi — three games with very different tactical character.

The same self-play-plus-search pattern has since influenced work well outside board games: MuZero extended it to Atari and other domains without even requiring known rules, learning its own internal model of the environment; the same core idea — pairing a learned evaluation with a search procedure over possible futures — informs planning algorithms in robotics and has conceptual echoes in how reasoning-focused language models use search-like strategies (sampling multiple reasoning paths and selecting the best) at inference time.

What's next

AlphaZero pairs a learned model with explicit search over a tree of possible futures — a pattern much closer to classical heuristic search than it might first appear. The Heuristics & Search in AI module covers that classical foundation directly, including A* search and the admissibility conditions MCTS's forward-looking evaluation builds on conceptually.