Cost Optimization for LLM Applications
Where the cost actually comes from
LLM API cost is almost always billed per token, split into input (prompt) and output (completion) tokens, with output tokens typically priced several times higher than input tokens because generation is more compute-intensive per token than reading context. Cost optimization therefore means reducing token volume, routing work to cheaper models where quality allows, and avoiding unnecessary round trips — not just picking a 'cheap model' and hoping accuracy holds.
Model routing and cascades
A model cascade sends every request first to a small, cheap, fast model; only requests it flags as uncertain, low-confidence, or out-of-scope get escalated to a larger, more expensive model. This exploits the fact that a large fraction of real-world queries are easy — simple lookups, well-templated classification, short factual answers — and don't need frontier-model capability. The hard part is building a reliable escalation trigger (confidence scoring, a cheap classifier, or self-reported uncertainty from the small model) that doesn't silently let hard cases fall through on the cheap path.
def answer_with_cascade(query, small_model, large_model, confidence_threshold=0.75):
small_result = small_model.generate(query, return_confidence=True)
if small_result.confidence >= confidence_threshold:
return small_result.text, "small_model", small_model.cost_per_1k_tokens
large_result = large_model.generate(query)
return large_result.text, "large_model", large_model.cost_per_1k_tokens
# Over a large volume of queries, if 70% resolve on the small model at
# 1/20th the per-token cost of the large one, blended cost drops roughly
# 0.7 * (1/20) + 0.3 * 1 ≈ 0.335x of the all-large-model baseline.Cost levers and their tradeoffs
| Lever | Mechanism | Risk |
|---|---|---|
| Prompt caching | Reuse cached prefix (system prompt, few-shot examples) at a discount | Only helps with stable, repeated prefixes |
| Model cascading/routing | Send easy queries to a cheaper model | Escalation logic can misclassify hard queries |
| Context trimming/compression | Summarize or truncate history before sending | Can lose information needed for the answer |
| Batching (async/offline workloads) | Use batch APIs at a lower per-token rate | Not viable for interactive, low-latency use cases |
| Output length limits | Cap max_tokens, use terse system prompts | Can truncate legitimately long answers |
| Smaller fine-tuned model for a narrow task | Replace a general large model with a specialized small one | Upfront fine-tuning cost and data requirement |
Prompt caching rewards stable prompt structure
Context length is a cost multiplier, not just a quality lever
Every token of retrieved context, conversation history, or few-shot examples is billed on every single call that includes it. A RAG pipeline that retrieves 10 chunks when 3 would suffice, or a chat app that resends the entire conversation history on every turn without summarization, pays for that redundancy at scale. Techniques like conversation summarization after N turns, aggressive reranking to retrieve fewer but higher-quality chunks, and chunk deduplication directly reduce per-request token cost in addition to often improving quality by removing noise.
Optimizing cost before measuring quality is backwards
What's Next?
Cost-aware engineering only works if prompts themselves are well-structured and reusable in the first place — which is exactly what disciplined prompt templating and few-shot formatting are designed to make possible.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.