How many examples each update uses: batch gradient descent uses the whole training set, stochastic gradient descent (SGD) one example, and mini-batch gradient descent a small batch, typically 32 to 512.
Card 1 of 277. Answer: How many examples each update uses: batch gradient descent uses the whole training set, stochastic gradient descent (SGD) one example, and mini-batch gradient descent a small batch, typically 32 to 512.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
All three are gradient descent: compute the gradient of the loss and step against it, w ← w − η g, with η the learning rate. They differ only in how many training examples go into g on each update. Batch (full-batch) gradient descent averages the gradient over all n examples, stochastic gradient descent (SGD) uses one randomly chosen example, and mini-batch gradient descent uses a batch of B examples, typically 32 to 512. Backpropagation is not one of them; it only computes the gradients.
The batch size sets how many updates an epoch buys. With 50,000 training images and a batch size of 256, an epoch takes 196 mini-batch updates, 195 full batches plus one of 80, versus one update for batch gradient descent and 50,000 for pure SGD. On ImageNet's 1,281,167 training images, one exact full-batch step costs as much compute as about 5,000 mini-batch steps of 256.
A single SGD step is cheap but noisy. Fit y = w·x with squared loss (w·x − y)², starting at w = 0. The example x = 2, y = 4 gives the gradient 2(w·x − y)·x = −16, so with η = 0.05 the weight jumps to 0.8 before the next example is drawn. Each such gradient is an unbiased estimate of the full gradient, and averaging B of them shrinks the noise like 1/√B: quadrupling the batch halves the noise but quadruples the work per step.
Batch gradient descent is deterministic, and on a smooth loss a small enough fixed rate lowers the loss on every step, reaching the global minimum if the loss is convex. Exact gradients also make quasi-Newton methods such as L-BFGS practical, which is what scikit-learn's LogisticRegression uses by default. Pure SGD zig-zags and needs a learning rate that shrinks over time to settle. Mini-batches sit in between and win in deep learning: they keep a GPU busy, never need the whole dataset in memory, and add noise that can help training move past saddle points.
Watch the vocabulary: in PyTorch, Keras and most papers, "SGD" means mini-batch SGD, not one example at a time. A common follow-up is what to change as the batch grows: scale the learning rate with it, linearly as a rule of thumb and with a warmup, and expect very large batches to sometimes generalize slightly worse. Memory does not force mini-batches either, since you can accumulate gradients over chunks and still take one exact full-batch step.
