A decoding method that keeps the k most probable partial sequences, the beam, at each step instead of only the single best next token. It finds higher-probability outputs than greedy decoding, which can commit early to a token that leads nowhere, at roughly k times the cost.
Card 1 of 292. Answer: A decoding method that keeps the k most probable partial sequences, the beam, at each step instead of only the single best next token. It finds higher-probability outputs than greedy decoding, which can commit early to a token that leads nowhere, at roughly k times the cost.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
At each step every sequence in the beam is extended by every possible next token, the candidates are scored by total log-probability, and the top k are kept. Greedy decoding is beam search with k = 1. Typical beams are 4 to 10 wide; wider beams cost more and often stop improving quality.
Example: the first token is A with probability 0.5 or B with 0.4. After A, the best next token has probability 0.4; after B, 0.9. Greedy decoding takes A and ends with 0.5 × 0.4 = 0.20. A beam of 2 keeps both A and B and finds B followed by its best token, 0.4 × 0.9 = 0.36.
Follow-ups: since each extra token multiplies in a probability below 1, beam search favors short outputs, so scores are normalized by length, as in Google's neural machine translation system (Wu and colleagues, 2016). For open-ended text it produces bland, repetitive output (Holtzman and colleagues, 2019, on neural text degeneration), which is why chat models sample with temperature and top-p instead. Beam search remains common in translation, speech recognition and other tasks with one right answer.
