An LSTM. The cell state carries information along a mostly additive path while the gates learn what to write, forget and read, which is how it survives the vanishing gradients that stall plain RNNs on long sequences. Attention has since displaced it for most sequence work.
Card 1 of 259. Answer: An LSTM. The cell state carries information along a mostly additive path while the gates learn what to write, forget and read, which is how it survives the vanishing gradients that stall plain RNNs on long sequences. Attention has since displaced it for most sequence work.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
An LSTM layer. The cell carries memory between steps. The forget gate sets how much of the old cell state to keep, the input gate how much new information to write, and the output gate how much of the cell to expose as the hidden state. Each gate is a sigmoid layer with outputs between 0 and 1.
Example: with old cell state 2.0, forget gate 0.9, input gate 0.5 and candidate −0.8, the new cell state is 0.9 × 2.0 + 0.5 × (−0.8) = 1.4. With output gate 0.7, the hidden state is 0.7 × tanh(1.4) ≈ 0.62. Since the cell updates by addition, information survives many steps while the forget gate stays near 1.
Interviewers like the parameter count. The three gates and the candidate each have weights, so 100 inputs and 256 units give 4 × (256 × (100 + 256) + 256) = 365,568 parameters, with one bias per block. Initializing the forget gate bias to 1 makes the cell remember by default.
