A Gated Recurrent Unit: a recurrent neural network cell with two gates, reset and update, and no separate cell state. It is a simpler alternative to the LSTM, with about 25% fewer parameters, and it usually performs about as well.
Card 1 of 267. Answer: A Gated Recurrent Unit: a recurrent neural network cell with two gates, reset and update, and no separate cell state. It is a simpler alternative to the LSTM, with about 25% fewer parameters, and it usually performs about as well.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
GRU stands for Gated Recurrent Unit, a recurrent neural network cell that uses gates to decide how much of its past state to keep at each time step. Kyunghyun Cho and colleagues proposed it in a 2014 EMNLP paper on encoder-decoder models for machine translation, as a simpler alternative to the LSTM.
It has two gates, each a sigmoid layer with outputs between 0 and 1. The reset gate rₜ controls how much of the previous state feeds the candidate state, h̃ₜ = tanh(W xₜ + U (rₜ ⊙ hₜ₋₁)); near 0, the candidate depends almost only on the current input. The update gate zₜ blends old and new, hₜ = (1 − zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ, doing the job of the LSTM's forget and input gates at once. Watch the convention: Cho and colleagues' paper multiplies the old state by z, many later sources by 1 − z.
Compared with an LSTM, a GRU merges the cell state and hidden state into one, so it needs no output gate: an LSTM's output gate chooses how much of its separate cell to expose, while a GRU exposes its whole state. With three weight blocks instead of four, it has about 25% fewer parameters than an LSTM of the same size. A layer with 100 inputs and 256 units has 3 × (256 × (100 + 256) + 256) = 274,176 parameters against the LSTM's 365,568, counting one bias per block.
The update gate is how a GRU holds memory: if it keeps 99% of a state each step, 0.99¹⁰⁰ ≈ 0.37 of it survives 100 steps. Which to pick is the usual follow-up. A 2014 comparison by Chung, Gulcehre, Cho and Bengio found the two comparable, and neither has won outright since, so in practice you try both. A GRU trains a little faster and is often preferred when data or compute is limited; an LSTM's separate cell sometimes helps on very long sequences. Both ease the vanishing gradient problem of plain RNNs, and Transformers have replaced both for most language tasks.
