Model-Based Reinforcement Learning

Two ways to get good at a task

There are two broad philosophies for solving an RL problem. Model-free methods, like the Q-learning and policy gradient methods covered in the reinforcement learning basics lesson, learn directly from trial and error without ever explicitly representing how the environment works — they just learn what actions tend to lead to reward. Model-based methods take an extra step: they learn an explicit model of the environment's dynamics — a function that predicts the next state (and often the reward) given the current state and action — and then use that model to plan, by mentally simulating possible futures before acting in the real world.

Why learn a model at all

The chief advantage of model-based RL is sample efficiency. Real-world or even simulated environment interactions can be expensive (think: a physical robot, or an expensive simulation step). Once an agent has a reasonably accurate model, it can generate as much additional 'experience' as it wants by querying the model instead of the real environment — essentially manufacturing cheap, simulated training data. This typically lets model-based methods reach good performance using far fewer real environment interactions than purely model-free alternatives, which must learn entirely from real experience.

Planning with a model

Once a model exists, there are several ways to use it. Model Predictive Control (MPC) repeatedly plans a short sequence of actions by simulating several candidate action sequences forward through the model, picks the one with the best predicted outcome, executes just the first action, observes the real result, and replans from scratch — constantly correcting for model inaccuracy by re-planning frequently. The Dyna architecture instead uses the model to generate extra simulated transitions that are fed into an otherwise ordinary model-free learning update, blending real and imagined experience in the same training loop.

Learning a model in a useful representation

Modern model-based systems often don't predict raw future observations (e.g. pixels) directly, since that's both hard and often unnecessary — a lot of pixel-level detail doesn't matter for decision-making. MuZero, for example, learns a model entirely in an abstract learned representation space, trained so that only the quantities that actually matter for planning — predicted reward, value, and policy — need to be accurate, without ever needing to predict what the next literal game frame will look like. This sidesteps a great deal of the model-error compounding problem, since the model only has to be right about what's decision-relevant.

Model-free vs. model-based RL

What's next

Model-based RL's planning loop pairs naturally with curiosity-driven exploration, which often reuses a learned forward model to measure surprise. Its reliance on a learned approximation of reality also mirrors the core challenge in sim-to-real transfer, just with a learned model standing in for a hand-built physics simulator.