Results From a GPU Cost Optimization Project, Numbers Included

A client's self-hosted inference bill had crept up to a number that finally got leadership's attention, and the ask was simple: find savings without touching output quality. Two weeks of focused work got the monthly GPU bill down by about 40%, and almost none of it required anything clever.

Where the waste actually was

Instances were provisioned at peak capacity around the clock, even though traffic had an obvious daily pattern with a quiet overnight window covering roughly a third of the day. Nobody had set up scheduled scaling because the original setup predated a reliable traffic pattern, and nobody revisited it once the pattern stabilized.

The biggest GPU cost savings weren't in the model, they were in the hours nobody was looking at the traffic graph.

What actually moved the number

• Scheduled scale-down during the predictable overnight low-traffic window, which alone accounted for roughly half the total savings.
• Request batching tuning that increased average batch size without increasing latency past the client's SLA, improving GPU utilization meaningfully.
• Right-sizing instance types — the team had defaulted to the largest available GPU tier out of caution during initial launch and never revisited it once load patterns were known.
• Spot/reserved instance mix for the predictable baseline load, keeping on-demand capacity only for traffic spikes.

Before: flat peak-capacity provisioning, 24/7
After:  scheduled scale-down (overnight) + right-sized instances
        + reserved baseline + on-demand burst
Result: ~40% monthly GPU spend reduction, 0% latency SLA impact

None of this touched the model, the prompts, or output quality — which was the whole point of the engagement. The uncomfortable finding, as usual, was that the savings had been sitting there the entire time; they just required someone to look at a cost dashboard with the explicit job of finding waste instead of glancing at it occasionally while doing something else.