Deep Learning
Gradient Descent is an algorithm used to minimize overfitting in neural networks.
False
Proposed by @svpino
Card 1 of 259. Answer: False
False
Proposed by @svpino
Card 1 of 259. Answer: False
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. Gradient descent is an optimization algorithm: it minimizes the training loss by repeatedly moving the weights against the gradient, w ← w − η ∇L(w). It has no notion of overfitting, which is fitting the training data, noise included, so closely that performance on new data suffers. Reducing overfitting is the job of regularization.
Run long enough, gradient descent can even cause overfitting. Fit a degree-9 polynomial to 10 noisy points: with 10 coefficients, the training error can be driven to zero by a curve threading every point and swinging wildly between them. Neural networks show it in their loss curves: training loss keeps falling while validation loss bottoms out and then rises.
Tools that fight overfitting include weight decay, dropout, data augmentation, more data and early stopping. Early stopping is the link to the optimizer: halting at the epoch with the best validation loss limits how far the weights travel, acting as a regularizer. Interviewers may also raise the implicit regularization of small-batch SGD, an open research question.