Hierarchical Reinforcement Learning
Long tasks need bigger building blocks
Imagine training an RL agent to make coffee by choosing individual motor torques at every millisecond. The task has thousands of steps, reward only arrives at the very end, and every tiny low-level action has to be learned from scratch. Humans don't plan at that level — we think in terms of 'grab the mug,' 'add water,' 'press brew,' each of which is itself made of lower-level motions. Hierarchical reinforcement learning (HRL) brings that same structure to RL agents: instead of choosing one primitive action at a time, the agent chooses among higher-level sub-goals or temporally extended behaviors, each of which runs for multiple timesteps before control returns to the higher level.
The options framework
The classic formalization of this idea is the options framework. An option is a temporally extended action: it has an initiation condition (when it's available), an internal policy (what to do while it's running), and a termination condition (when to hand control back). A high-level policy selects among options rather than primitive actions; each selected option then runs its own internal policy until it terminates, at which point the high-level policy chooses again. This is strictly more general than standard RL — a one-step option that always terminates immediately reduces to an ordinary primitive action.
Why this helps with sparse rewards and long horizons
Breaking a task into sub-goals attacks two problems at once. First, credit assignment becomes easier: instead of needing to propagate one distant reward signal back across thousands of primitive steps, each sub-goal can have its own, denser learning signal — reaching 'grab the mug' is itself a checkable milestone. Second, exploration becomes more efficient: exploring at the level of 'try a different sub-goal order' covers far more meaningfully different behavior than exploring at the level of random torque perturbations, because each sub-goal already encapsulates a competent chunk of behavior rather than needing to be discovered from scratch every time.
Note
Feudal reinforcement learning
A related architecture, feudal reinforcement learning, organizes this hierarchy explicitly like a chain of command: a 'manager' module operates at a slow time scale and sets abstract goals for a 'worker' module, which operates at a fast time scale and is rewarded purely for achieving the manager's goal — not the original task reward directly. The worker doesn't need to know or care what the ultimate task is, only how to satisfy whatever sub-goal it's handed, which lets the same worker be reused under many different managers or tasks.
Flat RL vs. hierarchical RL
| Aspect | Flat RL | Hierarchical RL |
|---|---|---|
| Decision granularity | One primitive action per environment step | Sub-goals/options spanning many steps, plus low-level execution |
| Credit assignment over long tasks | Hard — one distant signal spans many steps | Easier — each sub-goal can have its own signal |
| Exploration | At the level of primitive actions | At the level of meaningful sub-goal choices |
| Reusability | Policy tied to one specific task | Low-level skills often reusable across tasks |
What's next
Hierarchical RL is one of the strongest tools against the sparse rewards problem, by shortening the effective horizon reward has to travel. It also offers a promising structure for multi-agent reinforcement learning, where sub-goals can decompose a joint task into pieces individual agents can learn more independently.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.