Multi-Agent Debate & Consensus

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The idea: multiple model instances checking each other

A single LLM call produces a single answer with no internal mechanism to flag its own uncertainty or catch its own mistakes. Multi-agent debate runs several LLM instances (which can be copies of the same model, or genuinely different models) on the same question, has them see and critique each other's answers and reasoning across one or more rounds, and then aggregates toward a final answer — the hope being that errors one instance makes get surfaced and corrected by another instance that didn't make the same mistake, rather than everyone independently converging on the same blind spot.

The basic debate protocol

A typical setup: each of $k$ agent instances independently generates an initial answer and reasoning for the question. Then, for one or more rounds, every agent is shown the other agents' previous-round answers and reasoning, and asked to update its own answer in light of them — potentially sticking with its position if it finds the counterarguments unconvincing, or switching if persuaded. After a fixed number of rounds, a final answer is produced either by simple majority vote across the agents' final positions, or by a separate aggregator step (another LLM call, or a human) that synthesizes the debate transcript into one conclusion.

debate_round.py

python

def multi_agent_debate(question, agents, num_rounds=2):
    answers = [agent.answer(question) for agent in agents]

    for round_num in range(num_rounds):
        new_answers = []
        for i, agent in enumerate(agents):
            others_answers = [a for j, a in enumerate(answers) if j != i]
            prompt = f"Question: {question}\nOther agents answered: {others_answers}\n" \
                     f"Your previous answer: {answers[i]}\nRevise if you find their reasoning more convincing."
            new_answers.append(agent.answer(prompt))
        answers = new_answers

    return majority_vote(answers)  # or a separate aggregator/judge call

Debate helps most on tasks with a verifiable, checkable answer

Multi-agent debate tends to show the clearest gains on tasks where one agent's reasoning errors can be concretely pointed out and checked by another — arithmetic, logical reasoning chains, factual questions with a single correct answer. It's far less reliably helpful on open-ended, subjective, or creative tasks, where there's no ground truth for a 'debate win' to converge toward, and where social-conformity-like dynamics (agents converging on whichever position sounds most confident, independent of correctness) can appear instead of a genuine error-correction effect.

Multi-agent debate vs self-consistency (sampling) vs single-pass answer

Property	Single-pass answer	Self-consistency (sample+vote)	Multi-agent debate
Extra compute cost	1x	Nx (N independent samples)	Nx * rounds (agents see each other's output)
Error-correction mechanism	None	Only via majority agreement	Agents can be persuaded by explicit counterarguments
Benefits from model diversity	N/A	Limited (usually one model)	Can mix different models for diverse failure modes

Diminishing (and sometimes negative) returns with more rounds or agents

Empirically, most of the benefit from debate shows up within the first one or two rounds; additional rounds frequently show diminishing returns, and in some setups can even degrade accuracy if agents start converging toward a confidently-stated wrong answer simply because it was argued persuasively, a dynamic sometimes called 'sycophantic drift' in debate literature — one agent's confident but incorrect claim can pull others toward it rather than being corrected, especially when using the same underlying model for all debate participants, since they share the same blind spots and persuasion styles.

Debate multiplies cost and latency, and that needs to be justified

Running $k$ agents for $r$ rounds costs roughly $k \times r$ times a single LLM call, plus the aggregation step — a substantial multiplier that only pays for itself on tasks where the accuracy gain is large enough, and where latency isn't critical. For many production use cases, a cheaper technique (self-consistency sampling with majority vote, or a single well-prompted call with explicit chain-of-thought) captures much of the available accuracy improvement at a fraction of the cost, making debate worth reserving for genuinely high-stakes, reasoning-heavy decisions rather than applying by default.

What's Next?

Multi-agent debate is one of several ways of spending extra inference-time compute to improve reliability — it sits alongside self-consistency decoding and tree-of-thought search as different strategies for the same underlying tradeoff between answer quality and the compute budget you're willing to spend getting there.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →