Reward Shaping in Reinforcement Learning

Giving the agent some hints

Many real tasks only produce a meaningful reward at the very end — win or lose a game, reach the goal or don't. Learning from such sparse rewards can be painfully slow, because an agent acting randomly may almost never stumble into the one rewarding outcome. Reward shaping addresses this by adding designer-crafted intermediate rewards that guide the agent toward good behavior long before the final outcome is known — for example, rewarding a robot for reducing its distance to a goal, not just for touching it.

A simple example

Consider training an agent to navigate a maze where the only 'true' reward is +1 for reaching the exit and 0 everywhere else. With random exploration, the agent might wander for thousands of steps before ever reaching the exit by chance, giving it almost no learning signal. A shaped version might add a small reward proportional to the reduction in distance to the goal at every step. Now the agent gets frequent, informative feedback: moving closer to the exit is immediately rewarded, moving away is immediately penalized, long before it ever reaches the end.

Potential-based shaping: a principled way to do it

Adding arbitrary bonus rewards can change what the *optimal* policy actually is — the shaped task may no longer be the task you meant to solve. Potential-based reward shaping avoids this by defining a potential function $\Phi(s)$ over states (e.g. negative distance to goal) and adding a shaping term of the form $\gamma \Phi(s') - \Phi(s)$ to the reward. This specific form is provably guaranteed not to change the ordering of optimal policies — it only changes how fast the agent discovers them. It's one of the few reward-shaping techniques with a formal guarantee attached.

Designing shaped rewards more safely

A few practices reduce the risk of reward hacking: keep the true sparse reward in the mix rather than replacing it entirely with shaping terms, use potential-based shaping where possible for its theoretical guarantee, monitor trained policies for behaviors that maximize reward without visibly accomplishing the task, and prefer rewarding verified outcomes over easily-gamed proxies when feasible. None of these eliminate the risk — reward design remains more art than science — but they catch many of the most common failure modes.

Sparse reward vs. naive shaping vs. potential-based shaping

What's next

Reward shaping is one response to the sparse rewards problem; see that lesson for exploration-based alternatives like curiosity-driven exploration that don't require hand-crafting bonus rewards at all. For the deeper question of whether a reward function is even measuring the right thing, see inverse reinforcement learning.