Self-attention lets every token weigh every other token in one parallel step, instead of pushing information down a recurrent chain, so long-range dependencies survive and training parallelises. The cost is attention quadratic in sequence length.
Card 1 of 292. Answer: Self-attention lets every token weigh every other token in one parallel step, instead of pushing information down a recurrent chain, so long-range dependencies survive and training parallelises. The cost is attention quadratic in sequence length.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Transformers replaced RNNs because self-attention links any two tokens in one step and processes a whole sequence in parallel, so training scales on GPUs. Each token's query is scored against every key, and the softmax of the scores weights the values: softmax(QKᵀ/√dₖ)V.
An RNN computes hidden states one after another, so a signal from token 1 reaches token 100 through 99 updates. If each scales the gradient by 0.9, only 0.9⁹⁹ ≈ 3 × 10⁻⁵ survives; LSTM gates soften this but stay sequential. Per layer, attention costs about n²·d operations and recurrence n·d², so attention is even cheaper whenever sequence length n is below width d.
Attention predates the Transformer: Bahdanau, Cho and Bengio added it to an RNN translator in 2014; Vaswani and colleagues dropped the recurrence in 2017. Attention ignores order, so positions must be encoded. At 32,000 tokens each head computes about a billion scores per layer. Generation still runs token by token with a growing KV cache, while an RNN keeps a fixed-size state, so state-space models such as Mamba revisit recurrence.
