Optimizing KV-Cache Memory Footprint in High-Throughput LLM Serving

Executive Summary & Key Takeaways

• The Core Bottleneck: While training is compute-bound (FLOPs), large language model inference is memory bandwidth and VRAM capacity bound during autoregressive token generation.
• KV Cache Bloat: Storing past Key and Value vectors prevents redundant computation across attention heads, but scales linearly with sequence length ($O(N)$) and concurrent batch size ($B$).
• PagedAttention Solution: Borrowing concepts from operating system virtual memory paging eliminates internal and external VRAM fragmentation, lifting memory utilization from ~20-40% to over 96%.
• Quantization Impact: Migrating from FP16 to FP8 (E4M3/E5M2) or INT4 for KV cache tensors halves memory pressure with negligible perplexity degradation (<0.1%).

1. The Memory Wall in Autoregressive Inference

In modern decoder-only transformers (such as Llama 3, Mistral, and DeepSeek), token generation operates across two distinct phases: the prefill phase and the decode phase. While prefill processes the prompt tokens concurrently in parallel (compute bound), token decoding generates one token at a time autoregressively (memory bandwidth bound).

Without caching, predicting token $t+1$ would require recomputing self-attention matrices over all previous tokens $t_1, dots, t$. The Key-Value (KV) cache stores previously computed projection states in GPU High Bandwidth Memory (HBM), transforming an $O(N^2)$ operation into $O(1)$ computation per decode step.

2. Mathematical Foundations of KV Tensors

For a standard Multi-Head Attention (MHA) model with layer count $L$, attention head count $H$, head dimension $D$, sequence length $S$, batch size $B$, and numeric precision bytes $P$ (e.g. 2 bytes for FP16):

# Memory footprint calculation for KV cache
def calculate_kv_cache_bytes(layers, heads, head_dim, seq_len, batch_size, precision_bytes=2):
    # Factor of 2 accounts for storing both Keys and Values
    return 2 * layers * heads * head_dim * seq_len * batch_size * precision_bytes

# Example: 70B parameter model (80 layers, 64 heads, head_dim 128)
# Sequence length 4096 tokens, Batch size 16 in FP16
bytes_required = calculate_kv_cache_bytes(80, 64, 128, 4096, 16, 2)
print(f"Required VRAM: {bytes_required / (1024**3):.2f} GiB")
# Output: 160.00 GiB just for KV cache!

This explains why multi-tenant model servers experience sudden out-of-memory (OOM) errors even when model weights easily fit within GPU memory limits.

3. PagedAttention: Virtual Memory for GPUs

Prior to PagedAttention (popularized by vLLM), inference engines allocated contiguous chunks of GPU memory for each client session based on the maximum potential context length (e.g., reserving 4k or 8k tokens ahead of time). Because average conversations rarely reach maximum context limits, up to 70% of allocated GPU memory sat completely idle and fragmented.

PagedAttention introduces a page table abstraction inside CUDA kernels. Key and Value vectors are subdivided into fixed-size physical blocks (typically 16 or 32 tokens per block). Non-contiguous physical memory blocks are mapped seamlessly into logical sequence spaces:

4. FP8 vs INT4 KV-Cache Quantization

Beyond spatial management, numerical representation significantly reduces the byte size of each stored vector. Modern architectures (NVIDIA Hopper H100 and Blackwell B200) introduce native hardware acceleration for 8-bit floating point formats:

• FP8 (E4M3): 1 sign bit, 4 exponent bits, 3 mantissa bits. Ideal for values with tight dynamic ranges; retains high precision for attention weights.
• FP8 (E5M2): Matches standard IEEE FP16 dynamic range with reduced precision. Well suited for outlier-heavy gradients and activation scales.
• INT4 Group-Wise Quantization: Stores 4-bit integers with per-channel or per-block scaling factors. Cuts VRAM consumption by 75% compared to FP16, allowing massive concurrency on single workstations.

5. Production Benchmarks & Implementation

When implementing production inference serving pipelines, combine PagedAttention with Grouped-Query Attention (GQA) and Chunked Prefill. The following configuration demonstrates optimal production parameters for vLLM or custom Triton inference pipelines:

# Launching high-throughput serving daemon with optimized KV parameters
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \
  --block-size 16 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-chunked-prefill \
  --max-model-len 8192

6. Frequently Asked Questions

Verified Sources & Documentation

• paper Efficient Memory Management for Large Language Model Serving with PagedAttention — UC Berkeley / vLLM Project
• paper FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Stanford University
• doc NVIDIA TensorRT-LLM Architecture Specification — NVIDIA Developer Documentation