A GRU. It folds the LSTM's cell and hidden state into one and drops a gate, so it has fewer parameters and trains faster, usually matching LSTM accuracy on short and medium sequences.
Card 1 of 276. Answer: A GRU. It folds the LSTM's cell and hidden state into one and drops a gate, so it has fewer parameters and trains faster, usually matching LSTM accuracy on short and medium sequences.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
A GRU, or gated recurrent unit. Its two gates are the reset gate and the update gate; the output gate belongs to the LSTM. Each gate is a sigmoid layer whose 0-to-1 outputs scale how much of a signal passes.
The update gate sets how much of the old hidden state to keep versus replace with a new candidate, merging the LSTM's forget and input gates. The reset gate sets how much of the old state feeds that candidate; near 0, the candidate depends almost only on the current input. If the update gate keeps 99% of a state per step, 0.99¹⁰⁰ ≈ 0.37 survives 100 steps.
Why no output gate? An LSTM's output gate chooses how much of its separate cell state to expose. A GRU has only the hidden state and exposes all of it. Watch the convention: Cho and colleagues' original paper multiplies the old state by z, many later sources by 1 − z.
