ReLU, the rectified linear unit, max(0, x). It is cheap and its gradient is exactly 1 on the positive side, so it does not saturate as sigmoid and tanh do, which is much of why deep stacks became trainable. Its failure mode is dead units stuck at zero.
Card 1 of 259. Answer: ReLU, the rectified linear unit, max(0, x). It is cheap and its gradient is exactly 1 on the positive side, so it does not saturate as sigmoid and tanh do, which is much of why deep stacks became trainable. Its failure mode is dead units stuck at zero.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
ReLU, the rectified linear unit: f(x) = max(0, x). The question's rule is its definition: inputs at or below zero give 0, positive inputs pass through unchanged. It is cheap, needing only a comparison, and piecewise linear with a single kink at 0.
For the pre-activations [−2, 0, 0.5, 3], ReLU outputs [0, 0, 0.5, 3], and in backpropagation the local gradients are [0, 0, 1, 1]: active units pass the error back unchanged, inactive ones block it. The derivative at exactly 0 is undefined; PyTorch and TensorFlow use 0 there. Kunihiko Fukushima used rectifiers in 1969, but deep learning adopted them after Nair and Hinton (2010) and Glorot, Bordes and Bengio (2011).
A common follow-up is initialization. With zero-centered inputs a ReLU zeroes about half of them, so He initialization sets the weight variance to 2/nᵢₙ, double LeCun initialization's 1/nᵢₙ, which keeps the signal from shrinking with depth. Variants target weaknesses: Leaky ReLU for dead units, ReLU6 for low-precision mobile models, GELU in Transformers.
