For a fixed compute budget, a language model's size and its number of training tokens should grow in equal proportion, which works out to roughly 20 training tokens per parameter. Hoffmann and colleagues at DeepMind showed in 2022 that most large models of the time were too big and undertrained.
Card 1 of 292. Answer: For a fixed compute budget, a language model's size and its number of training tokens should grow in equal proportion, which works out to roughly 20 training tokens per parameter. Hoffmann and colleagues at DeepMind showed in 2022 that most large models of the time were too big and undertrained.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Scaling laws describe how loss falls as a power law in parameters, data and compute. Kaplan and colleagues at OpenAI (2020) concluded that extra compute should go mostly into bigger models. Hoffmann and colleagues trained over 400 models, from 70 million to more than 16 billion parameters on 5 to 500 billion tokens, and found that parameters and tokens should each grow with the square root of compute.
Training compute is about 6 × N × D floating-point operations for N parameters and D tokens. Gopher, 280B parameters on 300B tokens, used about 5.0 × 10²³. Chinchilla, 70B parameters on 1.4 trillion tokens (20 per parameter), used a similar 5.9 × 10²³, yet beat Gopher on most benchmarks, including MMLU, 67.5% against 60.0%, at a quarter of Gopher's size, which also makes it cheaper to serve.
The interview nuance: Chinchilla-optimal minimizes training compute, not total cost. A model served to millions of users pays for its size on every request, so it pays to train a smaller model far longer: Llama 3 8B saw 15 trillion tokens, about 1,900 per parameter. The open question now is data, since high-quality text is finite.
