How the next token is sampled: temperature flattens or sharpens the distribution, top-k keeps the k likeliest tokens, top-p the smallest set whose mass reaches p. Near zero for extraction and code; raise it only when you want variety.
Card 1 of 292. Answer: How the next token is sampled: temperature flattens or sharpens the distribution, top-k keeps the k likeliest tokens, top-p the smallest set whose mass reaches p. Near zero for extraction and code; raise it only when you want variety.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Temperature, top-k and top-p control how a language model picks each next token from its predicted probability distribution, trading predictability for variety. Temperature T rescales the logits, pᵢ = exp(zᵢ/T) / Σⱼ exp(zⱼ/T). Top-k keeps the k likeliest tokens; top-p, or nucleus sampling, keeps the smallest set whose probabilities reach p.
Example: after "The sky is", say the model gives blue 0.6, grey 0.3 and green 0.1. Dividing logits by T equals raising each probability to the power 1/T and renormalizing: T = 0.5 gives 0.78, 0.20 and 0.02, and T = 2 gives 0.47, 0.33 and 0.19. Top-p = 0.8 keeps blue and grey, as blue alone falls short, and renormalizes them to 0.67 and 0.33. Top-k = 1 is greedy decoding.
Interviewers ask why top-p usually beats top-k: a fixed k keeps junk when the distribution is peaked and cuts good tokens when it is flat; the nucleus adapts. Nucleus sampling comes from Holtzman et al., "The Curious Case of Neural Text Degeneration" (ICLR 2020). T = 0 means greedy, though GPU nondeterminism can still vary outputs slightly.
