Deep Learning
Leaky ReLU is an activation function that lets small negative values pass through when a neuron's input is less than zero.
True
Proposed by @svpino
Card 1 of 259. Answer: True
True
Proposed by @svpino
Card 1 of 259. Answer: True
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
True. Leaky ReLU passes positive inputs through unchanged and multiplies negative inputs by a small slope α instead of setting them to zero: f(x) = x if x > 0, otherwise αx. Strictly, the condition is on each neuron's own input, its weighted sum, not the network's input.
With α = 0.01, the inputs −5, −0.5 and 3 become −0.05, −0.005 and 3, where plain ReLU gives 0, 0 and 3. The gradient is the point: it is α, not 0, for negative inputs, so a neuron stuck in the negative region still gets small updates and can recover. Andrew Maas, Awni Hannun and Andrew Ng proposed it in 2013 for speech recognition networks.
Interviewers ask how to pick α. Defaults differ: PyTorch uses 0.01, the Keras layer 0.3, and DCGAN discriminators use 0.2. PReLU, from Kaiming He and colleagues in 2015, learns α during training. Leaky ReLU is as cheap as ReLU and avoids dead units, but gains are often small; transformers mostly use GELU or SiLU.