Backpropagation — reverse-mode automatic differentiation, which gets the gradient of a scalar loss with respect to every parameter in roughly one backward pass. Rumelhart, Hinton and Williams rediscovered and popularised it for neural networks in 1986.
Card 1 of 276. Answer: Backpropagation — reverse-mode automatic differentiation, which gets the gradient of a scalar loss with respect to every parameter in roughly one backward pass. Rumelhart, Hinton and Williams rediscovered and popularised it for neural networks in 1986.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Backpropagation. Linnainmaa's 1970 master's thesis at the University of Helsinki described reverse-mode automatic differentiation: computing the derivatives of a function built from elementary operations by applying the chain rule from the output back to the inputs. Backpropagation is this method applied to a neural network's loss.
He wrote the thesis in Finnish to study how rounding errors accumulate in numerical algorithms, with no mention of neural networks, and published the method in English in the journal BIT in 1976. Paul Werbos proposed using it to train neural networks in the 1970s and early 1980s, and Rumelhart, Hinton and Williams popularized it with their 1986 Nature paper "Learning representations by back-propagating errors".
Interviewers probe the cost. Reverse mode gives the gradient for every parameter in one backward pass costing a small constant multiple of the forward pass; forward mode needs one pass per input, so a million parameters would mean a million passes. The price is memory, since intermediate activations must be stored.
