Why Your GPU Runs Out of Memory with Local LLMs

The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs

SitePoint TeamPublished inAI·Computing·
August 1, 2026

SitePoint Premium
Stay Relevant and Grow Your Career in Tech

• Premium Results
• Publish articles on SitePoint
• Daily curated jobs
• Learning Paths
• Discounts to dev tools

7 Day Free Trial. Cancel Anytime.

You load a 7B parameter model onto a GPU. It fits comfortably. The first few prompts return without issue. Ten messages into the conversation, though, an out-of-memory error kills the process. This article breaks down exactly where every byte of VRAM goes during local LLM inference, provides a working Python calculator for estimating peak memory needs, and offers concrete techniques for managing KV cache growth.

Table of Contents

Prerequisites and version assumptions: The shell commands and feature references in this article were tested against llama.cpp (build b3600+), Ollama 0.5.x, and vLLM 0.6.x. Flag names and KV cache quantization support vary across versions. Verify flag availability with ./llama-server --help | grep cache-type (llama.cpp) or consult each project’s changelog before running commands. Python code blocks require Python 3.8+ and have no external dependencies.

The Mystery of the Disappearing VRAM

You load a 7B parameter model onto a GPU. It fits comfortably. The first few prompts return without issue. Ten messages into the conversation, though, an out-of-memory error kills the process. The model did not grow, and the GPU did not shrink, yet VRAM ran out. This scenario baffles developers running local LLMs every day. The culprit is the KV cache: a dynamic memory structure that silently consumes GPU memory with every token generated.

Model weights are static. They occupy a fixed footprint in VRAM from the moment they load. But memory consumption during inference is dynamic, driven primarily by the KV cache, which scales linearly with the length of the conversation. Most developers plan for the weight footprint and stop there. That gap between planning and reality is where OOM errors live.

Most developers plan for the weight footprint and stop there. That gap between planning and reality is where OOM errors live.

This article breaks down exactly where every byte of VRAM goes during local LLM inference, provides a working Python calculator for estimating peak memory needs, and offers concrete techniques for managing KV cache growth. It targets developers working with tools like llama.cpp, Ollama, vLLM, and text-generation-webui who understand the basics of model quantization but have not yet internalized the memory dynamics of long-context inference.

What Actually Lives in Your GPU Memory

Model Weights: The Fixed Cost

Model weights represent the learned parameters of the neural network. The parameter count and numerical precision together determine their VRAM footprint. A 7B parameter model stored in FP16 (2 bytes per parameter) occupies roughly 14 GB. The same model quantized to Q4_K_M (approximately 4.83 bits per parameter in practice) drops to approximately 4.2 GB for a 7B-parameter model (e.g., Mistral 7B) and approximately 4.8 GB for an 8B-parameter model (e.g., Llama 3 8B).

This is the part of the memory budget that developers typically understand and plan for. It also stays constant during inference.

The KV Cache: The Silent Memory Killer

The KV cache stores the key and value tensors that the attention mechanism produces for every token in the current context window. During autoregressive generation, each new token must attend to all previous tokens. Without a cache, the model would need to recompute the key and value projections for every prior token at every generation step, turning inference into an O(n²) operation per token. The KV cache trades memory for compute by storing these tensors so they can be reused.

The critical property: the KV cache grows linearly with the number of tokens in the context. At short conversations, it is negligible. At long contexts, it can exceed the size of the model weights themselves.

Activation Memory and Framework Overhead

Beyond weights and KV cache, GPU memory holds the CUDA context, which the driver allocates when you initialize the GPU. The inference framework also reserves scratch space for matrix operations and memory pools, and the forward pass produces intermediate activation tensors. This overhead typically ranges from 500 MB to 2 GB depending on the inference framework — llama.cpp tends toward the lower end, vLLM toward the higher end — and batch size. It is relatively stable but must be accounted for in any VRAM budget.

How the KV Cache Actually Works (Without the PhD)

Attention Needs Memory: A Visual Walkthrough

In self-attention, each token produces three vectors: a query, a key, and a value. The query for the current token is compared against the keys of all previous tokens to compute attention weights, which the mechanism then uses to compute a weighted sum of the corresponding values. The result is the “context-aware” representation of the current token.

The cache holds one key-value pair per head per layer at token 1, a thousand at token 1,000, and 32,000 at token 32,000. The model weights remain unchanged throughout. Picture two regions of GPU memory side by side: a fixed block for weights, and a buffer for the KV cache that expands rightward with every generated or ingested token.

The Math Behind KV Cache Size

The formula for KV cache memory consumption is:

KV cache (bytes) = 2 × num_layers × num_kv_heads × head_dim × context_length × dtype_bytes

The factor of 2 accounts for storing both keys and values. This formula assumes a single sequence (batch_size=1). For batch inference or multi-request serving (e.g., vLLM with --max-num-seqs > 1), multiply the result by the number of concurrent sequences.

Worked example: Llama 3 8B at FP16 with 8K context. Llama 3 8B uses 32 layers, 8 KV heads (grouped-query attention), and a head dimension of 128. At FP16 (2 bytes per value):

2 × 32 × 8 × 128 × 8,192 × 2 = 1,073,741,824 bytes ≈ 1 GB

The same model at 128K context:

2 × 32 × 8 × 128 × 131,072 × 2 = 17,179,869,184 bytes ≈ 16 GB

That is a 16x increase in KV cache memory for a 16x increase in context, and 16 GB of KV cache alone exceeds the capacity of most consumer GPUs.

The dramatic variance across models stems from architectural choices, particularly the number of KV heads. Qwen2.5 7B uses only 4 KV heads across 28 layers, resulting in roughly half the KV cache of Llama 3 8B at equivalent context lengths.

__all__ =["calculate_kv_cache_size"]defcalculate_kv_cache_size(num_layers:int,num_kv_heads:int,head_dim:int,context_length:int,dtype_bytes:float=2.0,verbose:bool=True,)->dict:"""Calculate KV cache memory for a given model architecture and context length.Assumes batch_size=1. For multi-request serving, multiply the resultby the number of concurrent sequences."""if dtype_bytes <=0:raise ValueError(f"dtype_bytes must be > 0, got{dtype_bytes}")ifany(v <=0for v in(num_layers, num_kv_heads, head_dim, context_length)):raise ValueError("num_layers, num_kv_heads, head_dim, context_length must all be > 0")kv_cache_bytes =(2* num_layers * num_kv_heads * head_dim * context_length * dtype_bytes)kv_cache_gb = kv_cache_bytes /(1024**3)if verbose:print("--- KV Cache Breakdown ---")print(f"Layers:{num_layers}")print(f"KV Heads:{num_kv_heads}")print(f"Head Dimension:{head_dim}")print(f"Context Length:{context_length:,}")print(f"Dtype Bytes:{dtype_bytes}")print(f"KV Cache Size:{kv_cache_gb:.2f}GB ({int(kv_cache_bytes):,}bytes)")print()return{"bytes":int(kv_cache_bytes),"gb": kv_cache_gb}if __name__ =="__main__":calculate_kv_cache_size(num_layers=32, num_kv_heads=8, head_dim=128, context_length=8192)calculate_kv_cache_size(num_layers=32, num_kv_heads=8, head_dim=128, context_length=131072)calculate_kv_cache_size(num_layers=28, num_kv_heads=4, head_dim=128, context_length=32768)

The Complete VRAM Budget Calculator

Adding It All Up: Weights + KV Cache + Overhead

The total VRAM requirement at peak usage follows this formula:

Total VRAM = model_weights + kv_cache(at target context) + overhead

Planning for startup memory is insufficient. The system must have headroom for the KV cache at the maximum context length the application will encounter. If a user sends a long document or a multi-turn conversation accumulates 16K tokens, the KV cache will grow to that size regardless of what fit at startup.

__all__ =["vram_budget"]MODEL_PRESETS ={"llama3-8b":{"params_b":8,"num_layers":32,"num_kv_heads":8,"head_dim":128},"mistral-7b":{"params_b":7,"num_layers":32,"num_kv_heads":8,"head_dim":128},"qwen2.5-7b":{"params_b":7,"num_layers":28,"num_kv_heads":4,"head_dim":128},"llama3-70b":{"params_b":70,"num_layers":80,"num_kv_heads":8,"head_dim":128},}QUANT_BPW ={"FP16":16,"Q8_0":8,"Q6_K":6.5,"Q5_K_M":5.5,"Q4_K_M":4.83,"Q4_0":4.5,"Q3_K_M":3.9,}defvram_budget(model_name:str,quant:str,context_length:int,available_vram_gb:float,kv_dtype_bytes:float=2.0,overhead_gb:float=1.5,verbose:bool=True,):"""Full VRAM budget calculator with max safe context estimation.Assumes batch_size=1. For multi-request serving (e.g., vLLM), multiplyKV cache by the number of concurrent sequences.Note: parameter counts use decimal (SI) billions; memory outputs usebinary GiB (1024^3). This introduces a ~7% underestimate of weightmemory. Treat results as approximate planning estimates."""if model_name notin MODEL_PRESETS:raise ValueError(f"Unknown model '{model_name}'. "f"Available:{sorted(MODEL_PRESETS.keys())}")if quant notin QUANT_BPW:raise ValueError(f"Unknown quantization '{quant}'. "f"Available:{sorted(QUANT_BPW.keys())}")if kv_dtype_bytes <=0:raise ValueError(f"kv_dtype_bytes must be > 0, got{kv_dtype_bytes}")if overhead_gb <0:raise ValueError(f"overhead_gb must be >= 0, got{overhead_gb}")model = MODEL_PRESETS[model_name]bpw = QUANT_BPW[quant]weight_bytes = model["params_b"]*1_000_000_000*(bpw /8)weight_gb = weight_bytes /(1024**3)kv_bytes =(2* model["num_layers"]* model["num_kv_heads"]* model["head_dim"]* context_length * kv_dtype_bytes)kv_gb = kv_bytes /(1024**3)total_gb = weight_gb + kv_gb + overhead_gbfits = total_gb <= available_vram_gbremaining_for_kv = available_vram_gb - weight_gb - overhead_gbbytes_per_token = kv_bytes / context_lengthmax_ctx =int(max(0, remaining_for_kv *(1024**3)/ bytes_per_token))if verbose:print(f"=== VRAM Budget:{model_name}@{quant}===")print(f"Model weights:{weight_gb:.2f}GB")print(f"KV cache ({context_length:,}ctx):{kv_gb:.2f}GB")print(f"Framework overhead:{overhead_gb:.2f}GB")print(f"Total required:{total_gb:.2f}GB")print(f"Available VRAM:{available_vram_gb:.1f}GB")print(f"Fits:{'YES'if fits else'NO -- will OOM at full context'}")print(f"Max safe context:{max_ctx:,}tokens")if max_ctx ==0:print("WARNING: Model + overhead already exceed available VRAM. ""Cannot run this configuration.")print()return{"total_gb": total_gb,"fits": fits,"max_context": max_ctx}if __name__ =="__main__":vram_budget("llama3-8b","Q4_K_M", context_length=32768, available_vram_gb=24.0)vram_budget("mistral-7b","Q4_K_M", context_length=16384, available_vram_gb=12.0)vram_budget("llama3-70b","Q4_K_M", context_length=8192, available_vram_gb=24.0)

Reference Table: Popular GPU and Model Combos

The table below provides approximate max practical context values computed from the calculator above. Because the calculator uses SI-to-binary mixed units and a fixed overhead estimate, treat these as planning estimates, not exact limits. Run the calculator with your own overhead and quantization values for precise results.

The numbers above assume approximately 1.5 GB of framework overhead and FP16 KV cache values. The most striking finding: a 7B Q4 model on a 12 GB GPU tops out around 14K to 28K context depending on architecture, nowhere near the 128K context length printed on the model card. A 70B model quantized to Q4_K_M barely fits on a 24 GB card with any meaningful context at all.

Why Context Length Claims Are Misleading

When a model card states “supports 128K context,” it means the developers trained the model with positional encodings and attention patterns that function at that sequence length. It says nothing about the memory required to actually serve that context.

Llama 3 8B at FP16 with 128K context requires approximately 16 GB of KV cache on top of 14 GB of model weights, totaling 30 GB before overhead — which exceeds 31 GB once even minimal framework costs are included. That surpasses the capacity of every consumer GPU currently on the market, including the RTX 4090 (24 GB).

The “context length” on a model card is an architectural property. The effective context window on a given machine is a hardware constraint. These are different numbers, and the smaller one always wins.

This gap between trained capability and deployable capability is a persistenthitectural property. The effective context window on a given machine is a hardware constraint. These are different numbers, and the smaller one always wins

Practical Techniques to Tame KV Cache Usage

Quantize the KV Cache (Not Just the Weights)

Most quantization discussions focus on model weights, but the KV cache itself can also be quantized. Storing key and value tensors in FP8 (1 byte) or Q8_0 (approximately 1 byte, with minor block-scale overhead) halves the KV cache memory compared to FP16. Going further to Q4_0 (0.5 bytes) cuts it to a quarter.

The quality trade-offs vary by precision level. Q8 KV quantization typically produces less than 0.1-0.3% perplexity increase in published benchmarks. Q4 KV quantization introduces measurable quality loss that remains acceptable for most conversational and coding tasks where verbatim recall is not required. However, Q4 KV cache can silently degrade outputs on tasks demanding precise recall over very long contexts (e.g., needle-in-a-haystack retrieval, multi-document QA); benchmark your specific use case before deploying Q4 in production.

Several inference engines support KV cache quantization directly:

./llama-server -m model.gguf --cache-type-k q8_0 --cache-type-v q8_0 -c32768./llama-server -m model.gguf --cache-type-k q4_0 --cache-type-v q4_0 -c65536

For Ollama users, context length can be set in a Modelfile:

FROM llama3:8b-q4_K_MPARAMETER num_ctx 32768

vLLM and ExLlamaV2 also support FP8 KV cache quantization. In vLLM, this is implemented as part of its paged attention backend; consult the vLLM documentation for your installed version to confirm flag names and supported quantization types.

Use Grouped-Query Attention (GQA) Models

When KV cache memory is your binding constraint, model selection matters more than runtime tuning. The KV cache formula includes num_kv_heads as a multiplier, and this varies dramatically across architectures.

Models using Multi-Head Attention (MHA) set the number of KV heads equal to query heads — typically 32 for 7B-class models like the original LLaMA 1. Models using Grouped-Query Attention (GQA) share KV heads across multiple query heads. Llama 3 8B uses 8 KV heads with 32 query heads, a 4:1 ratio that cuts KV cache to one quarter the size it would be under MHA.

This is a model selection decision, not a runtime configuration. Choosing a GQA model over an equivalently-sized MHA model provides a fourfold (or greater) reduction in cache memory with minimal quality impact. Most major 7B-70B open models released since 2023 (Llama 3, Mistral, Qwen2.5) use GQA. Older architectures like the original Llama 1 used MHA and carry proportionally larger KV caches.

Limit and Manage Context Length

The simplest mitigation is to set an explicit context limit lower than the model maximum. In llama.cpp, the -c flag controls this directly. In Ollama, the num_ctx parameter serves the same purpose. Setting -c 8192 on a model that supports 128K caps the KV cache at 1 GB (for Llama 3 8B at FP16) instead of letting it grow to 16 GB.

At the application layer, implementing sliding window context or context truncation prevents runaway KV cache growth in multi-turn conversations. When the conversation exceeds the configured limit, the oldest tokens are dropped.

Some model architectures handle this natively. Mistral 7B uses Sliding Window Attention (SWA) with a 4,096-token local window (per the original Mistral paper, 2023; verify against the sliding_window field in config.json for your specific model variant). The KV cache only holds keys and values for the most recent 4,096 tokens regardless of the total conversation length, bounding memory consumption at the cost of losing direct attention to earlier tokens.

Offload Strategically

When GPU VRAM is exhausted, you can offload the KV cache to system RAM. Both llama.cpp and vLLM support this. The trade-off is latency: every token generation step requires transferring relevant KV data from CPU memory to GPU over the PCIe bus. For long contexts, this increases per-token latency by 2x to 5x depending on PCIe generation (PCIe 3.0 at approximately 16 GB/s, PCIe 4.0 at approximately 32 GB/s per x16 slot) and context length.

KV cache offloading makes sense for batch processing or applications where latency is less critical than the ability to process long documents. For interactive chat applications, the latency penalty usually makes it impractical beyond modest context lengths.

Implementation Checklist: Before You Deploy a Local LLM

1. Calculate model weight size at your chosen quantization. Multiply parameter count by bits-per-weight, divide by 8 to get bytes. A 7B model at Q4_K_M (4.83 bpw) is approximately 4.2 GB; an 8B model at Q4_K_M is approximately 4.8 GB.
2. Your target context length is not the model’s maximum. It is the longest conversation or document your application will realistically handle. Set it accordingly.
3. Calculate KV cache at that context length using the formula. Use 2 × layers × kv_heads × head_dim × context_length × dtype_bytes. Use the calculator above or run it locally. For multi-request serving, multiply by the number of concurrent sequences.
4. Add 1 to 1.5 GB for framework overhead. CUDA context, inference engine buffers, and activation tensors occupy this space regardless of model or context.
5. Compare the total to your available VRAM. If total exceeds available memory, the system will OOM at peak context.
6. If it does not fit, apply mitigations in order of impact: reduce target context length, quantize the KV cache to Q8 or Q4, select a GQA model if using MHA, and offload KV cache to CPU as a last resort.
7. Test at peak context. Construct a prompt that fills the context window to the configured limit. Monitor VRAM usage continuously during generation: watch -n 0.5 nvidia-smi (Linux) or nvidia-smi dmon -s m -d 1. A one-shot nvidia-smi call may miss peak allocation.
8. Set explicit context limits in your serving configuration. Never rely on defaults or model maximums. Use -c in llama.cpp, num_ctx in Ollama, or --max-model-len in vLLM to enforce a ceiling that your hardware can sustain.

Calculating Your Actual Context Limit

The effective context window of a local LLM deployment is not determined by the model architecture. Available VRAM determines it, after accounting for weights, KV cache, and overhead. A model that “supports” 128K context but runs on a 12 GB GPU may be limited to 14K tokens in practice.

Running the calculator before selecting a model and GPU combination prevents the frustration of discovering memory limits during deployment. The relationship between context length and VRAM is linear, predictable, and calculable in advance.

Techniques like paged attention in vLLM (which manages the KV cache in fixed-size blocks to reduce fragmentation), ring attention for distributed inference (which distributes attention computation across devices for very long sequences), and ongoing research into KV cache compression reduce KV memory requirements further, though none eliminate the linear scaling described here. The fundamental mechanics — linear growth of key-value storage with sequence length — are inherent to the transformer attention mechanism. Understanding them is not optional for anyone deploying local LLMs under real memory constraints.

Sharing our passion for building incredible internet things.