Deep Reinforcement Learning (DRL) Overview

Reinforcement learning, briefly

Reinforcement learning (RL) is a framework for learning behavior through trial and error. An agent observes a state of its environment, takes an action, receives a reward, and transitions to a new state. The goal is to learn a policy — a mapping from states to actions — that maximizes cumulative reward over time. This is fundamentally different from supervised learning: there's no labeled dataset of correct actions, only a reward signal that the agent must use to figure out, through repeated experience, which actions tend to lead to good outcomes.

Why classical RL hit a wall

Classical RL algorithms like tabular Q-learning work by maintaining a table of values: for every state-action pair, a number estimating the expected future reward of taking that action in that state. This works fine when the number of states is small enough to enumerate — a grid world with a few hundred cells, say.

It completely breaks down for realistic problems. The raw pixel input of an Atari game has far more possible states than atoms in the universe; no table could ever hold a value for each one, and even if it could, the agent would never visit most states more than once, so it could never learn a value for them through direct experience. Real-world control problems need a way to generalize — to estimate the value of a state the agent has never seen, based on its similarity to states it has seen.

DQN: the breakthrough

The Deep Q-Network (DQN), introduced by DeepMind in 2013-2015, was the first algorithm to convincingly demonstrate that a neural network could learn to play Atari games directly from raw pixels, using only the game's score as a reward signal, and reach human-level performance on many of them.

DQN approximates the Q-function — the expected future reward of taking a given action in a given state — using a convolutional neural network that takes raw pixel frames as input and outputs a Q-value for every possible action. The action with the highest predicted value is (usually) chosen. Two techniques were critical to making this actually work, because naively training a neural network this way is highly unstable:

Experience replay. Instead of training on experiences in the order they occur (which are highly correlated, since consecutive game frames are nearly identical), DQN stores past experiences in a buffer and trains on randomly sampled batches from it. This breaks the correlation between consecutive training samples and lets each experience be reused multiple times.

Target network. The Q-learning update rule bootstraps — it uses the network's own current predictions to compute the training target for the next update. Using the exact same, constantly-changing network for both the prediction and the target makes training chase a moving target and can diverge. DQN fixes this by keeping a separate, slowly-updated copy of the network (the target network) to compute training targets, only periodically syncing it with the main network.

Tabular Q-learning vs Deep Q-Networks

What's next

DQN learns a value function and derives a policy from it implicitly (pick the action with the highest value). An entirely different family of algorithms skips the value function and optimizes a policy directly — the subject of the next lesson, policy gradient methods.