Information about each token's position, added to or mixed into its representation, because self-attention on its own treats the input as an unordered set. The original Transformer added fixed sine and cosine waves of different frequencies; most current LLMs use rotary position embeddings (RoPE) instead.
Card 1 of 292. Answer: Information about each token's position, added to or mixed into its representation, because self-attention on its own treats the input as an unordered set. The original Transformer added fixed sine and cosine waves of different frequencies; most current LLMs use rotary position embeddings (RoPE) instead.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Without positions, attention would see "dog bites man" and "man bites dog" as the same input. The sinusoidal encoding sets dimension 2i of position pos to sin(pos / 10000^(2i/d)) and dimension 2i + 1 to the matching cosine. Fixed encodings work at any position; learned position embeddings, as in BERT and GPT-2, stop at the longest position seen in training, 512 and 1,024 tokens.
Example: with d = 512, the first pair of dimensions repeats every 2π ≈ 6.3 positions and the last pair only every 60,000 or so, so together the waves give each position a distinct pattern, fine near the start and coarse further out. RoPE (Su and colleagues, 2021) instead rotates each pair of query and key dimensions by an angle proportional to the position, so the attention score between two tokens depends only on how far apart they are.
Follow-ups: ALiBi drops position embeddings and subtracts a penalty proportional to distance from the attention scores; models struggle with sequences longer than those seen in training, which is why methods such as position interpolation and YaRN rescale RoPE to extend the context window.
