Blog / Model guides
Gemma 4 31B VRAM requirements: why the weights fitting is not enough
Gemma 4 31B's BF16 weights can fit on an 80 GB GPU while the model server still refuses to start. That happened in our H100 deployment: loading succeeded, but the configured context needed more KV-cache memory than the engine had left.
The useful sizing question is how much memory remains for your requests after loading the exact checkpoint and runtime. Context length, concurrency and input type belong in that question from the start.
What our H100 deployment recorded
These figures come from the 24 August 2026 deployment record for a single Nebius H100 80 GB running vLLM 0.26.0 with the BF16 instruction model. The catalog serves this configuration as text-only. These are historical results for that setup, not a fresh benchmark of today's engines.
| Memory or setting | Recorded value |
|---|---|
| BF16 weight files | About 62.55 GB, or 58.25 GiB |
| Weights loaded into GPU memory | 57.91 GiB |
| Available KV cache at 90% memory utilization | 10.27 GiB |
| KV cache requested for a 32,768-token context | 15.79 GiB |
| Engine's estimated context ceiling | About 12,208 tokens |
| Context used for successful qualification | 8,192 tokens |
The 32,768-token setting exceeded the available cache by 5.52 GiB. The engine reported its own estimated ceiling; it is not a value to extrapolate to a different version or attention configuration. Reducing the configured context to 8,192 produced ready endpoints on RunPod and Nebius in the recorded qualification runs.
GB and GiB are different units: one GiB is about 1.074 GB. A weight-file total, the memory shown by a driver and a provider's GPU label should be converted to the same unit before comparing them.
The model's context limit is not your deployment's limit
Google documents a 256K context window for the larger Gemma 4 models. The 32,768-token value above was our attempted server setting, not the model family's full advertised context. The working 8,192-token value is a limit of the qualified deployment.
A request's context includes its prompt and generated continuation. Several simultaneous requests share the server's memory budget. Multimodal inputs can introduce additional allocations, so a text-only result does not establish the memory required for images or video.
What to try when vLLM reports insufficient KV cache
- Read the startup log and separate model weights, runtime allocations and available cache.
- Set a context limit that covers the actual task, including the expected answer.
- Start with low concurrency, then test representative long requests together.
- Evaluate a supported quantized checkpoint or a GPU with more memory if the useful workload still does not fit.
Increasing gpu_memory_utilization changes the engine's memory budget; it does not create physical memory. Leave room for the rest of the runtime and measure peak use. A server that starts successfully can still fail when an application sends a workload outside the one you tested.
Google's 4-bit server build is a separate option
The official gemma-4-31B-it-qat-w4a16-ct artifact uses quantization-aware training and compressed tensors. In the LLM Hangar catalog's pinned revision, the weight file is about 23.27 GB. The recorded 48 GB deployment uses a 32,768-token context and has a successful qualification dated 6 August 2026.
That is a specific tested configuration. It does not prove every 4-bit build fits the same hardware or that this setup serves the full 256K context. Google's format guide distinguishes server compressed-tensor builds from GGUF and unquantized QAT checkpoints. Download the artifact intended for your runtime.
For a useful comparison, run BF16 and the quantized build on the same extraction, reasoning or coding tasks. Check factual errors, formatting and accepted outputs. Smaller weights reduce the memory budget, but the quality tradeoff belongs to your workload.
Budget for the configuration you can actually use
A cheaper card with too little usable context is not a saving. Choose the required context and concurrency first, then compare hourly rates and scheduled hours for configurations that pass. The live table covers catalog shapes for this model; inspect the selected variant and context in the deployment configuration before treating a price as applicable.
Current prices, live
Questions people ask
Does Gemma 4 31B BF16 fit on an 80 GB GPU?
Our recorded H100 configuration loaded the weights but failed with a 32,768-token context because of insufficient KV cache. An 8,192-token text-only configuration qualified successfully. Other runtimes and workloads need their own verification.
Can the quantized model run on 48 GB?
The catalog records a successful 48 GB deployment of Google's w4a16 compressed-tensor build at 32,768 tokens. That result applies to the pinned artifact and configuration, not every quantized release or the full advertised context.
Does a successful boot prove production capacity?
No. Test the longest intended requests, simultaneous users, streaming behavior and peak memory. A boot establishes readiness for the tested configuration, not a throughput or availability guarantee.