January 1, 2026 · 5 min read
We Underestimated the Infra Complexity of Self-Hosting a Model
Standing up a self-hosted model felt like a weekend project in the planning doc and turned into six weeks of infrastructure work nobody had scoped, mostly around things that have nothing to do with the model itself.
The scoping doc said two weeks: stand up the model, wire it to the application, done. It took six. Almost none of the extra four weeks had anything to do with the model itself — inference worked on day three. Everything around it is what ate the schedule.
Where the time actually went
Autoscaling for a GPU workload behaves nothing like autoscaling for a stateless web service — cold starts on a loaded model are measured in tens of seconds, not milliseconds, so naive request-based scaling produced a wall of timeouts the moment traffic ticked up. We ended up needing predictive pre-scaling based on traffic patterns, which is its own small project.
Running inference was the easy 10%. Everything the model needed around it to behave reliably in production was the other 90% nobody scoped.
- Request batching had to be tuned by hand for our specific prompt-length distribution; the defaults were tuned for a different workload shape entirely.
- GPU driver and CUDA version mismatches between our dev and prod images cost three days that felt completely unrelated to the actual task.
- Observability was essentially nonexistent out of the box — we had to build custom metrics for queue depth and batch utilization from scratch.
- A memory leak in a long-running serving process, invisible at low traffic, surfaced only after a week of sustained load and took real effort to isolate.
None of this means self-hosting was the wrong call for that project — it wasn't, the long-term economics still worked out. But I now scope self-hosted model infrastructure the way I'd scope standing up a new piece of core platform infrastructure, not the way I'd scope swapping an API endpoint, because that's what it actually is.