A way to speed up LLM generation: a small draft model proposes several tokens, and the large model checks all of them in one forward pass, keeping the longest accepted run. With the right acceptance rule the output follows exactly the large model's distribution, so it gets faster with no loss in quality.
Card 1 of 292. Answer: A way to speed up LLM generation: a small draft model proposes several tokens, and the large model checks all of them in one forward pass, keeping the longest accepted run. With the right acceptance rule the output follows exactly the large model's distribution, so it gets faster with no loss in quality.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Decoding is limited by memory bandwidth, not arithmetic: each step reads all the weights to produce a single token, so scoring several proposed tokens in one pass costs about the same as generating one. Each draft token x is accepted with probability min(1, p(x) / q(x)), where p is the large model's probability and q the draft's; at the first rejection, a replacement token is sampled from the leftover distribution max(0, p − q), renormalized. Leviathan, Kalman and Matias at Google and Chen and colleagues at DeepMind published the method in 2022 and 2023.
Example: if the draft proposes 4 tokens and each is accepted with probability 0.8, the large model produces on average (1 − 0.8⁵) / (1 − 0.8) ≈ 3.36 tokens per forward pass instead of 1. The real speedup is lower because the draft model's time counts too; the papers report about 2 to 3 times.
It works best when text is predictable, as in code or low-temperature answers, because the acceptance rate drives the gain. Variants avoid a separate draft model: Medusa and EAGLE add small prediction heads to the large model, and prompt lookup decoding drafts by copying n-grams that already appear in the prompt.
