Text is cut into subword tokens from a fixed vocabulary; BPE builds that vocabulary from characters up, repeatedly merging the most frequent adjacent pair. So common words are one token, rare and non-English text cost more, and letters are invisible.
Card 1 of 292. Answer: Text is cut into subword tokens from a fixed vocabulary; BPE builds that vocabulary from characters up, repeatedly merging the most frequent adjacent pair. So common words are one token, rare and non-English text cost more, and letters are invisible.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Byte pair encoding builds a subword vocabulary by starting from characters or bytes and repeatedly merging the most frequent adjacent pair, until it reaches a target size. Philip Gage published BPE for compression in 1994; Sennrich, Haddow and Birch adapted it to machine translation in 2016.
Toy corpus: low ×5, now ×4, lot ×2. The commonest pair, o w, appears 9 times and becomes ow; then l ow (5) merges into low and n ow (4) into now. Lot stays l, o, t. Encoding replays the merges in order, so unseen town becomes t + ow + n. GPT-2 starts from all 256 bytes, so no input is unknown, and adds 50,000 merges and an end-of-text token: 50,257 tokens.
A bigger vocabulary shortens sequences but enlarges the embedding table: GPT-2 small's 50,257 × 768 matrix holds 38.6 million weights. The same text can need up to 15 times more tokens in one language than another (Petrov and colleagues, 2023). The model sees token IDs, not letters, so counting letters trips it up. BERT's WordPiece merges by likelihood instead.
