Deep Learning
Type of gradient descent where we use the whole dataset to perform a single update step.
Batch Gradient Descent.
Card 1 of 259. Answer: Batch Gradient Descent.
Batch Gradient Descent.
Card 1 of 259. Answer: Batch Gradient Descent.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Batch gradient descent, also called full-batch or vanilla gradient descent. Each update averages the gradient over all n training examples, w ← w − η (1/n) Σ ∇ℓᵢ(w), so one epoch buys exactly one step. Strictly, it is a variant of gradient descent; backpropagation only computes the gradients.
Because the gradient is exact, the run is deterministic, and on a smooth loss a small enough fixed rate lowers the loss on every step, reaching the global minimum if the loss is convex. Exact gradients also make quasi-Newton methods such as L-BFGS practical; scikit-learn's LogisticRegression uses it by default. The cost is per step: on ImageNet's 1,281,167 training images, one step costs as much compute as about 5,000 mini-batch updates of 256.
It suits small datasets and convex models; deep learning prefers mini-batches because one exact step is rarely worth thousands of noisy ones. Memory is not the real limit: you can accumulate the gradient over chunks and still take one exact step.