Deep Learning
Type of gradient descent where we use a fixed number of training examples, fewer than the whole dataset, to calculate the gradient.
Mini-batch Gradient Descent.
Card 1 of 276. Answer: Mini-batch Gradient Descent.
Mini-batch Gradient Descent.
Card 1 of 276. Answer: Mini-batch Gradient Descent.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Mini-batch gradient descent. Each update computes the gradient on a fixed-size batch of examples, typically 32 to 512, rather than the whole dataset (batch gradient descent) or one example (stochastic gradient descent). Strictly, these are variants of gradient descent; backpropagation only computes the gradient for each batch.
Example: with 50,000 images and a batch size of 256, an epoch takes 196 updates, 195 full batches plus one of 80, versus 1 for batch gradient descent and 50,000 for pure SGD. The batch gradient, (1/B) Σ ∇ℓᵢ, is an unbiased, noisy estimate of the full gradient; its noise shrinks like 1/√B, so quadrupling the batch halves the noise but quadruples the work.
Mini-batches win because they keep a GPU busy, never need the full dataset in memory, and add noise that can help training escape saddle points. A common follow-up: as the batch grows, scale the learning rate with it, and very large batches can generalize slightly worse. In PyTorch and Keras, "SGD" almost always means mini-batch SGD.