A Long Short-Term Memory network: a recurrent neural network with a cell state and three gates (forget, input and output) that decide what to erase, write and expose at each step, which lets it learn dependencies across long sequences.
Card 1 of 267. Answer: A Long Short-Term Memory network: a recurrent neural network with a cell state and three gates (forget, input and output) that decide what to erase, write and expose at each step, which lets it learn dependencies across long sequences.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
LSTM stands for Long Short-Term Memory, a recurrent neural network built to learn long-range dependencies in sequences. Sepp Hochreiter and Jürgen Schmidhuber introduced it in a 1997 Neural Computation paper of the same name. The name refers to making short-term memory, held in the network's activations rather than its weights, last for a long time.
Plain RNNs struggle with long gaps because gradients shrink or explode over many time steps. An LSTM adds a cell state that is updated mostly by addition, cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ, and outputs hₜ = oₜ ⊙ tanh(cₜ). The three gates are sigmoid layers with outputs between 0 and 1: the forget gate fₜ sets how much of the old cell state to keep, the input gate iₜ how much of the new candidate c̃ₜ to write, and the output gate oₜ how much of the cell to expose as the hidden state. The forget gate came later, added by Gers, Schmidhuber and Cummins in 1999.
A worked step: with old cell state 2.0, forget gate 0.9, input gate 0.5 and candidate −0.8, the new cell state is 0.9 × 2.0 + 0.5 × (−0.8) = 1.4. With output gate 0.7, the hidden state is 0.7 × tanh(1.4) ≈ 0.62. Because the cell updates by addition, information and gradients survive many steps while the forget gate stays near 1, which is why initializing the forget gate bias to 1 helps.
Interviewers like the parameter count. The three gates and the candidate each have their own weights, so a layer with 100 inputs and 256 units has 4 × (256 × (100 + 256) + 256) = 365,568 parameters, counting one bias per block (PyTorch keeps two, adding 1,024). Other common follow-ups: how an LSTM differs from a GRU, which has two gates and no separate cell state, and why Transformers, which process a whole sequence in parallel, replaced it in most NLP after 2017. LSTMs remain common for time series and on-device models.
