Training a smaller student model to imitate a larger teacher by matching the teacher's full output probabilities, not just the hard labels. The soft probabilities carry extra information, such as which wrong classes are close to the right one, so the student learns more from each example.
Card 1 of 292. Answer: Training a smaller student model to imitate a larger teacher by matching the teacher's full output probabilities, not just the hard labels. The soft probabilities carry extra information, such as which wrong classes are close to the right one, so the student learns more from each example.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Hinton, Vinyals and Dean described the method in 2015. Both models' logits are divided by a temperature T before the softmax, which spreads the probabilities out, and the student is trained on a mix of the usual loss on true labels and the KL divergence to the teacher's softened distribution, with the KL term multiplied by T² to keep its gradients in scale.
Example: a teacher's logits for cat, dog and car are 5, 3 and −2. At T = 1 its probabilities are 0.88, 0.12 and 0.001, close to a hard label. At T = 4 they become 0.56, 0.34 and 0.10, which shows the student that this image looks much more like a dog than a car. Hinton called this information dark knowledge.
Results: DistilBERT (Sanh and colleagues, 2019) is 40% smaller and 60% faster than BERT while keeping 97% of its language understanding. Many small LLMs are distilled, either from a larger model's token probabilities, as with Gemma 2's smaller models, or by training on text a larger model generated. Check the terms of service first, since many API providers forbid using outputs to train competing models.
