Deep Learning
What does gradient descent try to minimize?
A loss function.
Card 1 of 259. Answer: A loss function.
A loss function.
Card 1 of 259. Answer: A loss function.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Gradient descent minimizes a loss function, also called a cost or objective function: one number measuring how badly the model fits the training data, as a function of its parameters. Examples are mean squared error for regression and cross-entropy for classification.
Each step moves the parameters against the gradient, downhill. Example: fit y = w·x to the point x = 2, y = 6 with L(w) = (2w − 6)², whose derivative is 4(2w − 6). At w = 0 the gradient is −24, so a learning rate of 0.05 moves w to 1.2 and the loss falls from 36 to about 13. Further steps approach the minimum at w = 3.
A common follow-up: why not minimize accuracy? It is not differentiable, so classifiers minimize cross-entropy as a smooth stand-in. The loss often adds a regularization term, and mini-batch variants estimate the gradient from a few examples. For convex losses such as linear regression's, every local minimum is global; neural network losses are non-convex, with no such guarantee.