Training is slow: tiny steps need many more iterations, and on a fixed budget the model may stop before converging. With noisy mini-batch gradients it does settle closer to a minimum, but it can't step past a poor nearby minimum or a plateau.
Proposed by @Al_Grigor
Card 1 of 259. Answer: Training is slow: tiny steps need many more iterations, and on a fixed budget the model may stop before converging. With noisy mini-batch gradients it does settle closer to a minimum, but it can't step past a poor nearby minimum or a plateau.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Training gets slow. The learning rate η scales every update, w ← w − η∇L(w), so small steps mean many more iterations and, on a fixed budget, a run that may stop before converging. Its other effects, a more precise minimum and getting stuck, need qualifying.
The more precise minimum holds for noisy mini-batch training: SGD jitters around the minimum, and the jitter grows with η. For L(w) = ½w² with gradient noise of standard deviation σ, the weight settles with spread √(η/(2 − η))·σ: 0.23σ at η = 0.1, 0.07σ at η = 0.01. But the smaller rate closes the gap by 1% per step, not 10%, so schedules start high and decay.
A small rate does not create local minima. Small steps follow the slope into the nearest basin and lack the size and noise to hop out; larger steps can skip shallow dips. In deep networks the usual symptom is crawling across plateaus. Some studies link larger rates to flatter minima that generalize better, though this is debated.
