Offline Reinforcement Learning

Learning without ever touching the environment

Standard reinforcement learning assumes the agent can keep interacting with the environment — try an action, see what happens, adjust, repeat. In many real settings that's not possible: you can't let an untrained policy control a real patient's medication dosing, experiment on live financial trades, or crash real cars to learn from the crashes. Offline reinforcement learning (also called batch RL) learns a policy purely from a fixed, previously collected dataset of logged experience — no further interaction with the environment is allowed during training at all.

Why this sounds easy but isn't

At first this looks like ordinary supervised learning on logged transitions: you have a pile of (state, action, reward, next state) tuples, so just run your favorite RL algorithm on them, right? The problem is that standard RL algorithms like Q-learning rely on being able to query the environment for *any* state-action pair, including ones that weren't in the training data, whenever the algorithm's current estimate says that pair looks promising. Offline, there's no environment to query — only the fixed dataset, which only ever shows what the behavior policy that collected it actually did.

Conservative approaches: staying close to the data

Most practical offline RL algorithms address this by building in some form of conservatism — explicitly discouraging the policy from straying into state-action regions poorly covered by the dataset. Conservative Q-Learning (CQL) adds a penalty that pushes down Q-values for actions unlikely under the data-generating policy, rather than letting them float upward unchecked. Batch-Constrained Q-learning (BCQ) restricts action selection to actions a generative model judges plausible given the data. Other methods add an explicit penalty for policies that diverge too far from the behavior policy. All of these trade off some potential performance gain for the safety of not confidently extrapolating into the unknown.

Off-policy evaluation: knowing how good the policy is before deploying it

Because offline RL forbids trying the learned policy in the real environment during training, there's also no easy way to check how good it actually is before deployment — you can't just run it and see. Off-policy evaluation techniques try to estimate a candidate policy's expected performance using only the offline dataset, typically via importance sampling (reweighting logged transitions by how likely the new policy would have been to produce them) or by using a learned model of the environment. These estimates are themselves imperfect, but they're often the only signal available before a costly or risky real deployment.

Online RL vs. offline RL

What's next

Offline RL's core challenge — learning safely from fixed data rather than live interaction — pairs naturally with imitation learning, which makes a similar bet (learn from existing data, not exploration) via a simpler supervised objective. Sim-to-real transfer addresses a related constraint on getting enough safe real-world experience, just from the simulation side rather than the fixed-dataset side.