By ByteBot
Modal published engineering data this week showing they’ve cut GPU inference cold starts from roughly 2,000 seconds down to 50 seconds — a 40x improvement that changes the economics of serverless AI deployment. The numbers are backed by production data: over 15 million GPU snapshot restores tracked between February and April 2026, with vLLM boot times dropping from 95.7 seconds to 13.8 seconds mean and SGLang from 83.7 seconds to 17.5 seconds. This isn’t a synthetic benchmark. It’s measured across ten thousand real cold start events with CDFs that improve at every quantile.
Why GPU Inference Cold Starts Have Broken Serverless
A warm GPU instance serves the first inference token in roughly 30 milliseconds. A cold instance that’s scaled to zero takes 40 seconds or more. That’s a 1,000x latency gap, and it has defined what’s been possible with serverless GPU infrastructure for the last three years. Teams building user-facing AI applications — chatbots, voice assistants, real-time document processing — couldn’t tolerate 40-second waits, so they kept GPUs running 24/7. Reserved capacity. Always-on. Paying for idle hardware at 3 AM. According to independent production analysis, this gap has made scale-to-zero “impractical for applications where users expect immediate responses.”
The economics are straightforward but punishing. Serverless compute’s value proposition is billing only for active time. For GPU workloads, a 40-second cold start on an H100 costs real money per request at scale, eroding the entire cost advantage before the first token is generated. Replicate solved this by keeping pre-hosted models perpetually warm. RunPod addressed it with large pre-warmed instance pools. Both approaches punt on the underlying problem — they just shift the idle cost around. Modal went a different direction and attacked the latency directly.
Four Layers, Four Bottlenecks
Modal’s approach stacks four independent techniques, each addressing a different stage in the cold start sequence. According to Modal’s engineering blog published May 12, the baseline 2,000-second startup breaks down roughly as: instance allocation (~600s), container filesystem load (~300s), host-side initialization (~600s), and device-side GPU initialization (~500s). Their optimized path cuts each stage independently, with the four improvements compounding to reach ~50 seconds total.
Cloud buffer pools use a linear programming optimizer (Google’s GLOP solver) to maintain pre-warmed idle GPUs sized dynamically against real-time cloud pricing and observed supply. Instance allocation drops from 10-30 minutes to roughly 5 seconds. A custom FUSE filesystem built on libfuse loads only the container index (a few MB in under 100ms) and fetches individual files on-demand through a four-tier content-addressed cache — from page cache at sub-microsecond latency through local SSD to AZ cache servers to regional CDN. Container load drops from several minutes to about 15 seconds.
CPU checkpoint/restore uses gVisor’s runsc runtime to snapshot entire process state after initialization completes. New replicas restore from that snapshot, skipping Python import overhead entirely. The practical improvement is substantial: import torch — which triggers over 10,000 syscalls — drops from 30-60 seconds to 3-5 seconds. GPU checkpoint/restore goes further, using Nvidia’s driver-level CUDA checkpoint API to snapshot device memory to host memory. On restore, compiled kernels, CUDA graphs, and model weights come back from the snapshot instead of being rebuilt from scratch. This is the layer that moves vLLM from 95.7 seconds to 13.8 seconds mean and SGLang from 83.7 seconds to 17.5 seconds.
The techniques compound rather than add. GPU snapshots require host snapshots, which leverage the FUSE filesystem delivery, enabled by the cloud buffer infrastructure. Each layer removes a bottleneck that would otherwise dominate after the previous one is eliminated.
Production Numbers, Not Lab Benchmarks
Between February and April 2026, Modal tracked 35 million CPU snapshot restores and 15 million GPU snapshot restores across several hundred distinct organizations. The Reducto case study illustrates what this means in practice: a document processing platform handling enterprise-scale deadline-driven jobs needed to scale from zero to hundreds of GPUs within tens of minutes. GPU snapshots cut their cold starts six times over — from 70 seconds to 12 seconds — enabling genuinely serverless kilo-GPU workloads without idle reserved capacity sitting around waiting for deadline crunch.
The CDF data matters here. Averages are fine for marketing; distributions are what production teams care about. Modal’s measurements show improvement at every quantile — p50, p90, and p99 all benefit. A technique that improves the average while leaving the tail unchanged doesn’t solve the production problem. This one moves the whole distribution.
The Limitation That Matters
CUDA checkpoint/restore has one significant constraint: it only works on single-GPU setups. Multi-GPU workloads involving NCCL — the collective communication library used for tensor parallelism across multiple GPUs — cause deadlocks during restore. That rules out the GPU snapshot optimization for 70B+ parameter model inference, which almost always requires multi-GPU configurations. The vLLM RFC #34303 tracking native CUDA checkpoint integration explicitly flags this as the primary blocking issue.
However, the single-GPU constraint covers a large portion of production AI workloads that matter right now. Structured data extraction, vision-language models, audio and speech processing, and smaller language models in the 1-50GB range all run comfortably on single GPUs. For these workloads — which represent the majority of inference jobs at most organizations — the 40x improvement is available today. Frontier model serving at GPT-5 or Claude scale still requires always-on infrastructure. But that’s a narrower share of total AI inference than it was 18 months ago, and it’s getting narrower.
Key Takeaways
- Modal achieved 40x GPU cold start reduction (2,000s → 50s) through four compounding techniques: LP buffer pools, FUSE lazy filesystem, CPU checkpoint/restore, and GPU CUDA checkpoint/restore
- vLLM cold starts drop from 95.7s to 13.8s; SGLang from 83.7s to 17.5s — measured across 10,000+ real cold starts with production CDFs improving at every quantile
- 15 million GPU snapshot restores tracked in production between February and April 2026 across several hundred organizations — this is deployed infrastructure, not a research prototype
- The technique requires single-GPU configurations; NCCL deadlocks block multi-GPU workloads, so 70B+ model inference is not yet covered
- For structured extraction, audio, vision-language, and smaller LLMs, serverless GPU is now a credible production option — the economics no longer require always-on reserved capacity
