Policy Gradient Methods

A different way to think about the problem

DQN learns a value function and derives behavior from it indirectly: compute a Q-value for every action, then act greedily. Policy gradient methods skip the value function entirely and parameterize the policy directly as a neural network that outputs a probability distribution over actions given a state. Training adjusts the network's parameters to make good actions more probable and bad actions less probable, using gradient ascent directly on expected reward.

This reframing has real advantages: policies can naturally handle continuous action spaces (steering angle, joint torque) where picking the 'max' over an infinite set of actions, as DQN requires, isn't well-defined. Policies can also represent useful stochastic behavior — sometimes the optimal strategy genuinely involves randomness, like in rock-paper-scissors, which a value-based greedy policy can't express.

The objective and the gradient problem

The objective is simple to state: maximize the expected total reward obtained by following the policy. The hard part is computing a gradient of that objective with respect to the policy's parameters, because the reward depends on the environment's dynamics — which are typically unknown and not differentiable at all (you can't backpropagate through a physics engine or a game's reward function).

The policy gradient theorem resolves this with a clever identity, often called the log-derivative trick: it shows that the gradient of expected reward can be rewritten as an expectation over trajectories of (the reward obtained) multiplied by (the gradient of the log-probability of the actions taken). Crucially, this expression can be estimated using only samples — trajectories the agent actually experiences — without ever needing to differentiate through the environment itself.

REINFORCE: the simplest policy gradient algorithm

REINFORCE (1992) is the most direct implementation of the policy gradient theorem. The procedure: run the current policy to collect a full episode (a complete trajectory from start to end), compute the actual return obtained from each time step onward, and use those returns directly as the reward signal in the policy gradient update.

This is elegant and unbiased — it uses the true sampled return rather than an estimate — but it suffers from a serious practical problem: high variance. Because a single episode's outcome can swing wildly due to randomness unrelated to the quality of any individual action, the gradient estimate computed from one episode can be noisy enough that training requires an enormous number of episodes to converge reliably.

Reducing variance with a baseline

The standard fix is to subtract a baseline from the return before weighting the gradient: instead of multiplying the log-probability by the raw return R_t, multiply it by (R_t - b), where b is some estimate of the expected return from that state (often a learned value function). This quantity, return minus baseline, is an estimate of the advantage — how much better this particular action's outcome was compared to what was expected on average from that state.

Subtracting a well-chosen baseline doesn't change the expected value of the gradient (it remains unbiased, a fact provable from the math of the policy gradient theorem), but it substantially reduces its variance, because it centers the learning signal: actions that performed about as well as expected get pushed neither strongly up nor down, while only genuinely surprising outcomes drive large updates. A common and effective baseline is the state-value function V(s), learned alongside the policy — which is precisely the idea that leads into actor-critic methods.

REINFORCE without vs with a baseline

What's next

Learning a value function purely to serve as a variance-reducing baseline is a short step away from learning it as a full second component of the system. Actor-critic methods take that step, combining a policy (the actor) with a learned value function (the critic) that guides training at every step rather than only at the end of an episode.