Blog / Guides

LLM cold start time: where the minutes go

Published 12 October 2026 · Deployment observations from 24 August 2026; startup documentation checked 6 September 2026

The GPU can spend most of a deployment waiting for something else. Before it produces an answer, the provider has to allocate a machine, the runtime has to start, and the model files have to arrive. Buying a faster GPU may leave the slowest part untouched.

Our Gemma 4 31B deployment investigation on 24 August 2026 made that visible. Loading weights from disk into GPU memory took 12 seconds in the recorded Nebius H100 run. The runtime image pull took about seven minutes. On one attempt, provisioning alone took 29 minutes before the operating system booted.

Define where your cold-start clock begins

“Cold start” can mean a new cloud instance, a new container on existing hardware, or an engine waking while its process is still alive. Those are different operations. Record the starting state along with the time.

vLLM Sleep Mode, for example, releases much of a model's GPU memory while keeping the server process alive. It does not itself stop the rented cloud instance. Its wake time should not be compared directly with provisioning a new GPU machine.

The recorded startup phases

The following observations come from the August investigation of Gemma 4 31B BF16 on one Nebius H100 in eu-north1, using vLLM 0.26.0. They are phase timings from the investigation, not a claim that every attempt reached a ready endpoint. Some operations overlap, so adding the rows does not produce a reliable end-to-end total.

Separate successful qualification records for the 8,192-token configuration show approximately 14 minutes from create to ready on RunPod and 20 minutes on Nebius that day. These observations illustrate the work involved; they are too limited to rank the providers or establish a typical startup time.

Optimize the phase that is actually slow

If allocation dominates, a smaller image will not fix the delay. Check provider capacity and the alternative locations or shapes your application permits. If downloads dominate, keep a verified copy of weights near the deployment and avoid rebuilding or downloading dependencies at every start.

Modal's cold-start guidance also distinguishes container startup from initialization work such as loading large files. That distinction is useful beyond Modal: cache hits, parallel loading and application initialization need separate measurements.

Staging weights removes a repeat download from the original repository, but the next instance must still read those bytes. Storage bandwidth, CPU work, GPU transfers and compilation can become the next bottleneck. Pin the model revision and runtime so a comparison is not accidentally measuring a different artifact.

Measure readiness from the client's point of view

An open port is an early milestone. For an application, readiness means an authenticated request can return a valid answer. Record both that first successful response and the first streamed token if interactive latency matters.

1. Capture timestamps for the request, allocation, OS boot, downloads, engine load and first successful inference.
2. Record model revision, engine image, hardware, region and cache state.
3. Repeat fresh and staged starts at the hours you actually need them.
4. Keep failed attempts in the results and record their cleanup time.
5. Use the observed spread to choose a startup allowance and a maximum wait.

For example, a hypothetical 15-minute billed startup at $2/hour costs $0.50. Repeating it on 22 weekdays adds $11 to compute cost. Storage may be worth paying for if it consistently reduces that startup, but verify the saving before calling it a bargain.

Make long starts manageable

Set a readiness timeout, a spending limit and a clear action when startup fails. LLM Hangar's staging and resume controls retain model files when configured, and its budget controls bound deployment attempts. Provider billing and the platform's cost estimate are separate records; reconcile them when measuring startup cost.

For a morning service, start early enough to cover the measured delay and tell users when capacity is unavailable. A schedule can initiate a launch; it cannot make unavailable GPUs appear.

Questions people ask