A sophisticated gradient descent algorithm that rescales the gradients of each parameter, effectively giving each parameter an independent learning rate.
Card 1 of 259. Answer: A sophisticated gradient descent algorithm that rescales the gradients of each parameter, effectively giving each parameter an independent learning rate.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
AdaGrad, short for adaptive gradient, is a gradient descent variant that gives each parameter its own learning rate. It divides each parameter's step by the square root of the sum of its past squared gradients, so frequently updated parameters slow down while rare ones keep taking large steps. John Duchi, Elad Hazan and Yoram Singer published it in 2011.
The update is Gᵢ ← Gᵢ + gᵢ², then θᵢ ← θᵢ − η gᵢ / (√Gᵢ + ε). Say every nonzero gradient is 1. A common word's weight, updated every step, has Gᵢ = 100 after 100 steps, so its step is η/10. A rare word's weight, updated for the first time, has Gᵢ = 1 and moves a full η. That suits sparse features; the GloVe word vectors were trained with AdaGrad.
Its weakness: Gᵢ only grows, so the effective learning rate decays toward zero and deep networks can stall. RMSProp swaps the sum for an exponential moving average, so old gradients fade, and Adam adds momentum on top.
