An LLM Fallback Strategy Saved Us During a Provider Outage

We built a fallback path to a secondary LLM provider for a client's production feature mostly as a precaution — single points of failure make me nervous, and at the time nobody had actually experienced an outage bad enough to justify it. Four months later, the primary provider had a multi-hour outage during peak business hours.

What actually happened

Our monitoring flagged elevated error rates within about ninety seconds of the outage starting. The circuit breaker tripped automatically after a threshold of consecutive failures, rerouting traffic to the secondary provider without a human needing to intervene. The feature stayed up the entire outage, with a brief quality dip since the fallback model wasn't our first choice, but functioning.

The fallback path cost us a few days of extra engineering we couldn't fully justify at the time. It paid that back in about three hours during an outage nobody could have scheduled around.

What made it actually work under pressure

• The circuit breaker logic was tested in a drill beforehand, deliberately simulating a provider outage, which is the only reason we trusted it to trigger automatically instead of needing a manual flip during the real thing.
• Prompt templates were written to be provider-agnostic where possible, so the switch didn't also require a prompt rewrite under pressure.
• We'd accepted a known quality tradeoff on the fallback model ahead of time, as a business decision, instead of discovering the tradeoff mid-outage and panicking about it.

The honest accounting is that this project's ROI looked bad for four months and then looked extremely good for one afternoon. That's the normal shape of resilience work — it's a cost you pay repeatedly for a payoff that arrives once, unpredictably, and is very easy to defer until the outage makes the decision for you.