ReLU, max(0, x), zeroes negative inputs and passes positive ones through. Unlike sigmoid it doesn't saturate for positive inputs, where its gradient is 1, so it greatly reduces vanishing gradients in deep networks, and it is cheaper to compute.
Proposed by @Al_Grigor
Card 1 of 276. Answer: ReLU, max(0, x), zeroes negative inputs and passes positive ones through. Unlike sigmoid it doesn't saturate for positive inputs, where its gradient is 1, so it greatly reduces vanishing gradients in deep networks, and it is cheaper to compute.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
ReLU, the rectified linear unit, is f(x) = max(0, x): negative inputs become 0 and positive ones pass through. It beats sigmoid in hidden layers mainly because it does not saturate for positive inputs, where its gradient is exactly 1, so gradients survive many layers. It is also cheaper, with no exponential.
Sigmoid's derivative, σ(x)(1 − σ(x)), is at most 0.25 and about 0.0066 at x = 5. Backpropagation multiplies one such factor per layer; over ten sigmoid layers they alone multiply to at most 0.25¹⁰ ≈ 9.5 × 10⁻⁷, while an active ReLU's factor is 1. In the 2012 AlexNet paper, a four-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than with tanh.
ReLU reduces vanishing gradients rather than solving them. A unit whose input stays negative gets zero gradient and can die, ReLU outputs are not zero-centered, and deep networks also need He initialization, normalization and residual connections. Follow-up: why did zero-centered tanh also lose? It saturates at both ends, like sigmoid.
