A layer in which each token builds its new representation as a weighted average of every token's value vector, with the weights set by how well its query matches their keys: softmax(QKᵀ / √dₖ)V. Each token can draw on any other token directly, whatever the distance between them.
Card 1 of 292. Answer: A layer in which each token builds its new representation as a weighted average of every token's value vector, with the weights set by how well its query matches their keys: softmax(QKᵀ / √dₖ)V. Each token can draw on any other token directly, whatever the distance between them.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Each token's vector is multiplied by three learned matrices to give a query, a key and a value. A token's query is compared with every key by a dot product, the scores are divided by √dₖ and passed through a softmax so they sum to 1, and the output is the weighted sum of the values. It is called self-attention because queries, keys and values all come from the same sequence; in cross-attention the queries come from one sequence and the keys and values from another, as when a decoder attends to an encoder. Decoders add a causal mask so a token can't attend to later ones.
Example: one query scores 4, 2 and 0 against three keys with dₖ = 4. Dividing by √4 = 2 gives 2, 1 and 0, and the softmax turns those into weights of 0.67, 0.24 and 0.09. The √dₖ scaling matters because dot products grow with dimension, and large scores push the softmax into a region where gradients almost vanish.
Follow-ups: compute and memory grow with the square of the sequence length; attention by itself ignores word order, so positions must be encoded; and Transformers run several attention heads in parallel, multi-head attention.
