Attention keys and values for the prefix, cached per layer, so each new token attends over saved state instead of recomputing the sequence. The cache grows with batch size, sequence length and layers, so GPU memory rather than compute limits throughput.
Card 1 of 292. Answer: Attention keys and values for the prefix, cached per layer, so each new token attends over saved state instead of recomputing the sequence. The cache grows with batch size, sequence length and layers, so GPU memory rather than compute limits throughput.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The KV cache stores the attention keys and values of every processed token, so each new token attends over stored state instead of re-running the prefix. It caps concurrency because each request holds its own cache in GPU memory, which runs out before compute.
Per token: 2 (keys and values) × layers × KV heads × head dimension × bytes. Llama 2 7B has 32 layers and 32 KV heads of dimension 128, so at 16 bits: 2 × 32 × 32 × 128 × 2 = 524,288 bytes, 2 GiB per 4,096-token sequence. An 80 GB GPU, after 13.5 GB of weights, fits at most about 30 of them.
Why not cache queries? A past token's query is never reused. Decoding is memory-bound: each step reads the weights and every cache to emit one token per sequence, so throughput grows with batch size until caches fill memory. Fixes: grouped-query attention, which gives Llama 3 8B a quarter of the cache with 8 KV heads; 8-bit cache quantization; and PagedAttention, which allocates the cache in blocks to curb fragmentation.
