Open-Source vs Managed LLM, a Real Cost Comparison After Switching

The case for self-hosting open-source models usually gets made on per-token cost alone, which flatters self-hosting because it ignores everything that isn't the token. We ran a steady, high-volume classification workload on both setups for two months each to get an honest comparison.

The number everyone quotes

On raw per-token pricing, self-hosting an open-source model on reserved GPU instances came out to roughly a third of the managed API's cost at our volume. That's the number that gets cited in every 'switch to open source' hot take, and at face value it's accurate.

The per-token price comparison is real, and it's also the least important number in the decision if your team isn't already running GPU infrastructure.

The numbers nobody quotes

• Engineering time to stand up and tune the serving infrastructure (vLLM, batching, autoscaling) ate about three weeks that the managed API never required.
• On-call burden increased measurably — GPU instance failures and OOM crashes needed a human at 2am in a way an API outage, which is the vendor's problem, didn't.
• Reserved GPU capacity meant paying for idle headroom during low-traffic hours, which the managed API's pay-per-token model never charged us for.
• Model quality on our specific task was about 4% lower on the open-source model, which mattered more on edge cases than the aggregate accuracy number suggested.

When we added engineering time and on-call cost to the open-source side's ledger, the total cost gap narrowed to something like 15-20% in self-hosting's favor — real, but nowhere near the 3x the headline number implied. We kept the managed API for this client, because they didn't have spare infrastructure capacity and the smaller savings didn't justify taking on GPU ops as a new standing responsibility. For a client who already runs GPU infrastructure for other reasons, the calculus flips entirely — the fixed cost is already paid.