Deep Learning
Stochastic Gradient Descent is the proper technique to use when the full dataset doesn’t fit in memory.
True
Proposed by @svpino
Card 1 of 259. Answer: True
True
Proposed by @svpino
Card 1 of 259. Answer: True
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
True. Stochastic gradient descent, and in practice its mini-batch form, is the standard way to train when the data does not fit in memory. Each update needs only one example or one small batch, so the rest of the dataset can stay on disk and be streamed in, an approach called out-of-core learning.
The numbers make the point. ImageNet's 1.28 million training images, decoded to 224 × 224 × 3 arrays of 32-bit floats, would take about 770 GB, while a batch of 256 takes about 154 MB. Data loaders read one batch at a time and prepare the next while the GPU works.
The nuance: full-batch gradient descent does not strictly need the data in memory either, since you can stream chunks and sum their gradients, but then every update costs a full pass over the data. In scikit-learn, SGDClassifier and SGDRegressor train out of core through partial_fit, one chunk at a time, whereas a standard kernel SVM solver needs the whole training set in memory.