Deep Learning
Dropout is an excellent regularization technique that speeds the training process.
False
Proposed by @svpino
Card 1 of 259. Answer: False
False
Proposed by @svpino
Card 1 of 259. Answer: False
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. Dropout is a widely used regularizer, but it slows training down. Each step trains a different random sub-network, so gradients are noisier and the network needs more updates to converge, while each update costs about the same.
Srivastava et al., in their 2014 JMLR paper, wrote that a dropout network typically takes 2 to 3 times longer to train than a standard network of the same architecture, naming noisy parameter updates as a major cause. So a model that peaks after 20 epochs might need 40 to 60 with dropout. Implementations save no compute per step either: dropped units are still computed, then multiplied by zero.
The trade pays off when a model overfits, since the slower run usually ends at a lower test error. The technique that does speed training is batch normalization: Ioffe and Szegedy matched an ImageNet model's accuracy in 14 times fewer training steps. A related trap: training loss can sit above validation loss, because dropout is active only during training.