Multi-Armed Bandits Are Overkill for Most Small-Traffic Sites

A client once asked me to set up a multi-armed bandit test for a landing page getting about 400 visitors a week. The pitch they'd gotten from a tool vendor was that bandits are strictly better than A/B tests because they minimize regret — they shift traffic toward the winning variant in real time instead of waiting for a fixed test to conclude. That's true. It's also almost entirely irrelevant at 400 visitors a week.

What bandits actually trade off

A bandit's advantage is real when traffic volume is high enough that the 'exploration cost' of running a clean A/B test — showing the losing variant to half your traffic for the full test duration — is meaningfully expensive in absolute terms. At low traffic, that exploration cost is already small, and what you give up with a bandit is something that matters more at small scale: a clean, interpretable, statistically rigorous answer to whether the result was real. Bandits are harder to reason about, harder to explain to a stakeholder, and in a low-traffic setting take longer to converge on a confident winner than a well-powered fixed test would.

A multi-armed bandit solves the problem of wasted traffic during a test. At low traffic, you don't have enough traffic to waste in the first place, and you do have a real need for a result you can actually trust.

A better heuristic than the vendor's pitch

• If your test would take more than a few months to reach significance with a standard fixed design, your traffic is probably too low for a bandit to meaningfully help either — the problem is traffic volume, not the testing method.
• If stakeholders need to understand and defend the result to someone outside the team, a fixed A/B test with a clear p-value story is far easier to communicate than a bandit's converged allocation.
• Reserve bandits for genuinely high-traffic, high-stakes scenarios — many simultaneous variants, real exploration cost, and a team that can interpret the output correctly.
• For most small sites, a simple, well-powered, properly-durationed A/B test is not the less sophisticated option. It's the correctly sized one.

We ran a standard fixed-duration test instead. It took six weeks to reach a confident answer, which felt slow to the client at first, but it was a real answer they could act on and explain, which is worth more than a sophisticated method that would have taken just as long to converge and produced a result nobody on the team could confidently interpret.