How LLM GPU memory is calculated
GPU memory for running a model breaks down into four parts:
- Weights:
parameters × bits per weight ÷ 8. An 8B model is about 15 GB in BF16, 7.5 GB in FP8, and 4.6 GB in GGUF Q4_K_M (≈4.89 effective bits, including quantization scales). This tool reports binary gigabytes (GiB), which is what nvidia-smi shows. - KV cache:
2 × layers × KV heads × head_dim × bytes × context × concurrent sequences. Grouped-query attention (fewer KV heads than attention heads) is why modern models have much smaller KV caches than older ones. - Activations: small for inference. For training they scale with batch × sequence length × hidden size × layers, which gradient checkpointing reduces.
- Overhead: CUDA context, allocator fragmentation, and framework buffers, usually 1–2 GB plus a few percent.
Choosing a quantization
Q8_0 and FP8 are nearly lossless for most models. Q5_K_M and Q4_K_M are the usual choices for local inference, cutting memory by about 3× compared with BF16 with a small quality loss. Below 4 bits (Q3, Q2), quality drops noticeably, especially for smaller models. For serving on GPUs, AWQ, GPTQ, and FP8 work well with vLLM and TensorRT-LLM. See our quantization guide for details.
Serving many users: the KV cache is the real limit
For a production agent or chat app, the weights are a fixed cost but the KV cache grows with every concurrent conversation. Serving 32 users at 8K context on Llama 3.1 8B needs 32 GB of KV cache in FP16, twice the size of the weights. Inference engines like vLLM use paged attention and prefix caching to share memory between requests, and FP8 KV cache halves it again.