During training, each unit in a layer is set to zero independently with probability p on every forward pass, so each gradient step trains a different random sub-network. At inference no units are dropped.
Card 1 of 259. Answer: During training, each unit in a layer is set to zero independently with probability p on every forward pass, so each gradient step trains a different random sub-network. At inference no units are dropped.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Dropout randomly switches off units during training. On every forward pass, each unit in a layer has its output set to zero independently with probability p, the dropout rate, so each gradient step trains a different random sub-network. Nothing is dropped at inference. Hinton and colleagues introduced it in 2012.
So the number dropped is random, not fixed: in a layer of 100 units with p = 0.5, about 50 are zeroed per step, perhaps 46 on one step and 55 on the next. Survivors are multiplied by 1/(1 − p), here 2, so the next layer's expected input matches what it sees at test time. This is inverted dropout, used by PyTorch and Keras.
It works because units cannot rely on particular other units, and the weight-sharing sub-networks act like a cheap ensemble. A classic bug is forgetting model.eval() in PyTorch, which leaves dropout on at inference. Watch the convention: the 2014 JMLR paper by Srivastava et al. uses p for the probability of keeping a unit.
