Can accelerate the training. However, it is possible that we “shoot” too far and miss the minimum of the function that we want to optimize.
Proposed by @Al_Grigor
Card 1 of 259. Answer: Can accelerate the training. However, it is possible that we “shoot” too far and miss the minimum of the function that we want to optimize.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
When the learning rate is too large, each gradient descent step overshoots the minimum. The loss bounces around instead of settling, and if the rate is large enough it grows from step to step until it diverges, often ending in NaN values.
The learning rate scales how far the weights move along the negative gradient. Near a minimum the right step is small; a step that is too big jumps to the other side of the valley, sometimes higher up than where it started. Symptoms are a loss curve that oscillates wildly, spikes, or climbs, and in deep networks exploding activations or gradients.
The usual fixes are to lower the rate by a factor of 3 to 10, add a warmup period so early steps are small, use a decay schedule, or use an adaptive optimizer like Adam. A learning-rate range test, which raises the rate each batch and plots the loss, is a quick way to find the largest rate that still trains stably.
