Curiosity-Driven Exploration
Motivation that doesn't come from the task
When extrinsic task rewards are sparse — as discussed in the sparse rewards problem lesson — an agent can spend most of its time wandering with no useful feedback at all. Curiosity-driven exploration addresses this by giving the agent an additional, self-generated reward signal: an intrinsic reward for encountering novel or surprising states, independent of whatever the actual task reward is. The idea borrows loosely from how curiosity works in animals and children — we're drawn to explore things that are unfamiliar or unpredictable, not only things we know are rewarding.
Prediction error as a curiosity signal
The most common way to operationalize curiosity is prediction error. The agent maintains a learned model that tries to predict what will happen next — typically the next state, or a compressed representation of it — given the current state and action. When the model's prediction turns out to be wrong, that's treated as a sign the agent has found something unfamiliar or hard to predict, worth exploring further, and an intrinsic reward proportional to the prediction error is added on top of (or instead of) the task reward.
Note
Fixing it: predicting in a learned feature space
A more robust fix, used in methods like Intrinsic Curiosity Module (ICM) and Random Network Distillation (RND), is to make predictions in a *learned, compact feature space* rather than raw pixel space, and to train that feature space to ignore aspects of the state the agent's own actions can't influence or that aren't task-relevant. ICM trains an inverse model (predict the action taken, given state and next state) as a side objective, which forces the feature representation to encode only what's controllable by the agent — static noise the agent can't affect gets filtered out, largely sidestepping the noisy TV problem. RND instead compares predictions against a fixed random network's output, which is deterministic and learnable in principle, so true randomness doesn't fool it the same way.
Count-based curiosity as a simpler alternative
A simpler, cheaper family of intrinsic rewards just tracks how often each state (or a hashed/compressed version of it) has been visited, and rewards the agent more for visiting rarely-seen states — directly rewarding novelty without needing a predictive model at all. This count-based approach is easier to implement and doesn't suffer from the noisy TV problem the same way, but it scales poorly to very large or continuous state spaces where almost every state is technically 'new', which is why prediction-error methods remain popular for complex, high-dimensional environments.
Approaches to intrinsic motivation
| Method | Signal used | Vulnerable to noisy TV problem? |
|---|---|---|
| Raw prediction error | Error predicting raw next state | Yes |
| Count-based novelty | Visitation counts of states | No, but scales poorly to huge state spaces |
| ICM (feature-space, action-predictive) | Prediction error in a controllability-filtered feature space | Largely avoided |
| RND (random network distillation) | Error predicting a fixed random network's output | Largely avoided |
What's next
Curiosity-driven exploration is one of several answers to the sparse rewards problem; see that lesson for how it fits alongside reward shaping and hierarchical approaches. It's also commonly combined with model-based reinforcement learning, since the forward model curiosity relies on can double as a planning model.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.