Ordinary reinforcement learning assumes someone has already written down a reward function, and the agent's job is to find the policy that maximizes it. But for many real tasks — driving politely, folding laundry the way a person would, writing in a particular style — nobody can cleanly specify the reward function in the first place. Inverse reinforcement learning (IRL) flips the problem: given observed behavior from an expert, infer the reward function that explains it. Once you have that reward function, you can hand it to ordinary RL, or simply understand what the expert was actually optimizing for.

Why not just copy the actions?

You might ask: if you have expert demonstrations, why not just train a policy to copy the expert's actions directly? That's imitation learning, and it's a valid, often simpler alternative (covered in its own lesson). IRL is more ambitious: it recovers the underlying *reason* the expert acted that way. That reward function can generalize to situations the expert never demonstrated — if you understand that a driver values both speed and safety, you can predict reasonable behavior in a novel intersection, whereas a cloned policy might fail outside its training distribution.

Note

IRL is fundamentally ill-posed: many different reward functions can rationalize the same observed behavior. A policy that always goes straight home could be explained by 'minimize travel time,' 'minimize fuel use,' or 'avoid unfamiliar streets' — and infinitely many reward functions make any fixed policy look optimal (including the trivial reward of zero everywhere). Every practical IRL method has to add some assumption or regularizer to pick a sensible reward function out of this huge ambiguous set.

Maximum entropy IRL

The most influential practical answer to that ambiguity is maximum entropy IRL. Instead of assuming the expert is perfectly optimal, it assumes the expert is *more likely* to take high-reward trajectories, but not deterministically — behavior is modeled as a probability distribution over trajectories, with probability proportional to the exponential of that trajectory's total reward. Among all reward functions consistent with the data, maximum entropy IRL picks the one that makes the observed behavior likely while otherwise assuming as little as possible (maximum entropy) about anything unobserved. This also naturally captures the fact that real experts are noisy and occasionally suboptimal.

python

Apprenticeship learning and feature matching

A related, simpler framing is apprenticeship learning: rather than recovering the exact reward, find a policy whose long-run feature statistics (e.g. average speed, distance from obstacles, fuel used) match the expert's, assuming the true reward is some unknown linear combination of those features. This sidesteps fully solving the reward ambiguity problem — it guarantees the learned policy performs at least as well as the expert *under the expert's own true reward function*, whatever that turns out to be, as long as the features are well chosen.

Note

IRL's core question — 'what reward is this behavior actually optimizing?' — is directly relevant to AI safety and alignment. Techniques descended from IRL underpin research into inferring human preferences from behavior rather than from hand-written reward functions, which is exactly the problem RLHF tries to solve at the level of chat preferences rather than physical demonstrations.

Inverse RL vs. imitation learning vs. standard RL

Approach	Input	What it produces	Generalizes beyond demos?
Standard RL	Hand-written reward function	A policy	Yes, within the given reward
Imitation learning	Expert demonstrations (state-action pairs)	A policy that mimics the expert	Limited — struggles off-distribution
Inverse RL	Expert demonstrations	An inferred reward function (+ optionally a policy)	Yes — reward can explain novel situations

What's next

For the simpler, more direct alternative to recovering a reward function, see the imitation learning lesson, which covers behavioral cloning and DAgger. IRL's reward-design lens also pairs naturally with reward shaping, which looks at the risks of hand-crafted reward functions from the other direction.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →