Standard reinforcement learning assumes a single agent acting in a fixed, if unknown, environment. Multi-agent reinforcement learning (MARL) drops that assumption: multiple agents act simultaneously, and each one is part of every other agent's environment. A self-driving car negotiating a four-way stop, two companies competing on price, or a team of warehouse robots coordinating pickups are all multi-agent problems. The moment more than one learner is present, the clean theory of single-agent RL starts to crack.

Cooperative, competitive, and mixed settings

MARL problems fall along a spectrum. In fully cooperative settings, all agents share a single team reward and succeed or fail together — think of a soccer team of robots. In fully competitive (zero-sum) settings, one agent's gain is another's loss, as in chess or poker. Most real problems are mixed-motive: agents partially cooperate and partially compete, like autonomous vehicles that want to avoid collisions (shared interest) while each minimizing its own travel time (competing interest). The right algorithm and reward structure depend heavily on which regime you're in.

Note

Single-agent RL convergence proofs rely on the environment being stationary — the same action in the same state always leads to the same distribution of outcomes. With multiple learning agents, that assumption fails: as other agents update their policies, the effective environment any one agent sees keeps shifting underneath it. An action that was optimal last episode may be exploited or obsolete this episode because another agent adapted. This is called non-stationarity, and it means Q-learning's convergence guarantees simply don't carry over.

text

Centralized training, decentralized execution

A widely used pattern sidesteps some of the non-stationarity problem during learning. In centralized training, decentralized execution (CTDE), each agent's critic (value estimator) is given access to the observations and actions of all agents during training, which makes the training signal stationary again — the critic can see the full picture. At execution time, however, each agent's policy (actor) only uses its own local observation, so the system still works in a decentralized, realistic deployment. Algorithms like MADDPG (Multi-Agent DDPG) use exactly this split.

Self-play: becoming your own training partner

In competitive two-player settings, a remarkably effective trick is self-play: an agent trains against copies of itself (often from earlier checkpoints), and as it improves, so does its opponent, creating an automatic curriculum of increasing difficulty. This is how systems like AlphaGo and OpenAI Five reached superhuman performance without any human-generated training data — the agent's own growing skill supplied an endlessly scaling challenge. Self-play can also produce instability if the agent overfits to its most recent self, which is why pools of past checkpoints are often sampled as opponents rather than only the latest version.

python

Note

In single-agent RL the goal is a single optimal policy. In competitive multi-agent RL, the right target concept often comes from game theory: a Nash equilibrium, a set of policies where no agent can improve by unilaterally changing its own strategy, given what everyone else is doing. There may be no single 'best' policy independent of what opponents do — only mutually consistent ones.

Single-agent RL vs. multi-agent RL

Aspect	Single-agent RL	Multi-agent RL
Environment dynamics	Stationary, fixed by the world	Non-stationary — shifts as other agents learn
Target solution	A single optimal policy	Often an equilibrium among policies
Training signal	Agent's own reward only	May need team reward or opponent-aware critics
Common technique	Q-learning, PPO	CTDE (e.g. MADDPG), self-play

What's next

MARL's non-stationarity problem is closely related to the sparse rewards problem, since coordinating agents often only get feedback at the end of a long joint episode — see the sparse rewards lesson. It also connects to hierarchical reinforcement learning, where sub-goals can decompose a joint task into pieces that are easier for each agent to learn independently.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →