Is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.
Card 1 of 259. Answer: Is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The learning rate is the hyperparameter that sets how big a step gradient descent takes on each update: new weights = old weights − learning rate × gradient. It is usually written η or α, and it is often the single most important hyperparameter to tune.
Too small, and training crawls and can stall on plateaus or in poor local minima. Too large, and the updates overshoot the minimum, so the loss oscillates or diverges. Typical starting values are 0.1 to 0.01 for SGD and around 0.001 (3e-4 is a popular default) for Adam.
In practice the rate is rarely constant. Schedules such as step decay, cosine annealing or warmup followed by decay change it during training, and adaptive optimizers like Adam and RMSProp scale it per parameter. A learning-rate range test, which raises the rate every batch and watches the loss, is a quick way to find a good value.
