Actor-Critic Methods

Combining the two families of RL algorithm

Value-based methods like DQN learn a value function and derive a policy from it. Policy gradient methods like REINFORCE learn a policy directly but suffer from high-variance gradient estimates, especially when the learning signal is a return only available at the end of a full episode. Actor-critic methods combine both ideas: a policy network (the actor) decides what action to take, and a separate value network (the critic) estimates how good the current state (or state-action pair) is, providing a lower-variance, step-by-step learning signal that trains the actor far more efficiently than waiting for whole-episode returns.

The actor and the critic, and how they interact

The actor is a policy network, exactly like in REINFORCE — it maps states to a distribution over actions, and is updated to make high-advantage actions more likely.

The critic is a value network, trained with ordinary supervised-style regression to predict the expected return from a given state, similar to the value function used as a baseline in the previous lesson. The key difference from a static baseline is that the critic is learned and updated *continuously*, in tandem with the actor, and it can be used to provide a learning signal after every single step rather than only at the end of an episode.

This is the crucial efficiency gain: REINFORCE has to wait for an episode to finish before it has any return to learn from. Actor-critic methods use the critic to estimate an advantage at every time step using only one step of real reward plus the critic's own prediction of the remaining value — a technique called bootstrapping, the same idea DQN uses internally. This lets the actor update far more frequently and with much less noisy gradients.

A2C and A3C: parallelizing experience collection

A2C (Advantage Actor-Critic) is the synchronous, standard implementation of the idea above, typically run with multiple parallel environment copies to collect a batch of experience before each update — improving stability by averaging the advantage signal over several simultaneous trajectories rather than relying on one noisy sequence.

A3C (Asynchronous Advantage Actor-Critic), its predecessor, took a different route to the same goal: many worker processes each run their own copy of the environment and their own copy of the actor-critic network, asynchronously pushing gradient updates to a shared set of parameters whenever they're ready. This both diversifies the experience being learned from (different workers naturally explore different parts of the state space at any given moment, which helps avoid the correlated-sample problem) and parallelizes training across CPU cores without needing a GPU or an experience replay buffer at all.

A2C later became more popular than A3C in practice, largely because the synchronous version is easier to implement correctly and get full benefit from on GPU hardware, while delivering comparable performance.

REINFORCE vs Actor-Critic vs A2C/A3C

What's next

Actor-critic methods round out the core toolkit of value-based and policy-based reinforcement learning. Stepping back from RL, a parallel revolution was happening in how models learn useful representations without any labeled data or reward signal at all -- the subject of the next lesson, self-supervised learning.