Deep Learning
Type of gradient descent where we use only a single training example to calculate the gradient and update the parameters.
Stochastic Gradient Descent.
Card 1 of 259. Answer: Stochastic Gradient Descent.
Stochastic Gradient Descent.
Card 1 of 259. Answer: Stochastic Gradient Descent.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Stochastic gradient descent (SGD). In its original form it computes the gradient of the loss on one randomly chosen training example and updates the parameters immediately: w ← w − η ∇Lᵢ(w), with η the learning rate. Strictly it is a variant of gradient descent, not of backpropagation, which only computes the gradient.
Example: fit y = w·x with squared loss (w·x − y)², starting at w = 0. The example x = 2, y = 4 gives the gradient 2(w·x − y)·x = −16, so with η = 0.05 the weight moves to 0.8 before the next example is drawn. With a million examples, one epoch makes a million updates, where batch gradient descent makes one.
Each single-example gradient is a noisy but unbiased estimate of the full gradient, so progress zig-zags and the learning rate must shrink over time for SGD to settle. Watch the vocabulary: in PyTorch and most papers, "SGD" means mini-batch SGD, with batches of 32 to 512 examples, which use a GPU far better.