Running several attention operations in parallel, each with its own learned projections of the queries, keys and values into a smaller subspace, then concatenating the results and projecting them back. Different heads can track different relationships, such as nearby words, syntax or what a pronoun refers to, within the same layer.
Card 1 of 292. Answer: Running several attention operations in parallel, each with its own learned projections of the queries, keys and values into a smaller subspace, then concatenating the results and projecting them back. Different heads can track different relationships, such as nearby words, syntax or what a pronoun refers to, within the same layer.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
With model width d and h heads, each head works in d / h dimensions, so the total cost is about the same as one full-width head. The outputs of the heads are concatenated and multiplied by an output matrix W_O. The original Transformer (Vaswani and colleagues, 2017) used 8 heads with a model width of 512, so 64 dimensions per head.
Example: in that setting the query, key, value and output matrices are each 512 × 512, so the attention block has 4 × 262,144 = 1,048,576 weights, plus biases. That number is the same with 1 head or 8; the heads change how the dimensions are split, not the parameter count.
Why not one big head? A single softmax gives each token one weighted average, so it can't attend strongly to two places for two different reasons at once. Studies of BERT found heads that specialize, for instance in direct objects or coreference (Clark and colleagues, 2019), but also that many heads can be pruned with little loss (Michel and colleagues, 2019). Grouped-query and multi-query attention keep many query heads but share keys and values to shrink the KV cache.
