An attention variant in which groups of query heads share one key head and one value head. It shrinks the KV cache, and the memory traffic of generating each token, by the group size while keeping quality close to full multi-head attention.
Card 1 of 292. Answer: An attention variant in which groups of query heads share one key head and one value head. It shrinks the KV cache, and the memory traffic of generating each token, by the group size while keeping quality close to full multi-head attention.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Multi-head attention gives every query head its own keys and values; multi-query attention (Shazeer, 2019) shares a single key-value head across all of them, which is fastest but loses some quality. Grouped-query attention, from Ainslie and colleagues at Google (2023), sits in between: with 64 query heads and 8 key-value heads, each group of 8 query heads shares one. They also showed an existing multi-head model can be converted by averaging its heads and training briefly, about 5% of the original pretraining compute.
Example: Llama 2 70B has 80 layers, 64 query heads, 8 key-value heads and a head dimension of 128. At 16-bit precision its cache needs 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, 1.25 GiB for a 4,096-token sequence. With 64 key-value heads it would need 10 GiB, eight times as much.
Why share keys and values but not queries? Only keys and values are cached for later tokens, and reading the cache is what limits decoding speed. Most open LLMs now use it, Llama 3 and Mistral among them. DeepSeek-V2's multi-head latent attention goes further, compressing keys and values into a small latent vector.
