Deep Learning
ReLU is an activation function that, given an input vector, generates an output where the sum of the values in the vector is equal to one.
False
Proposed by @svpino
Card 1 of 259. Answer: False
False
Proposed by @svpino
Card 1 of 259. Answer: False
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. That describes softmax, not ReLU. ReLU, the rectified linear unit, is applied to each element on its own: ReLU(x) = max(0, x). Negative values become 0 and positive values pass through unchanged, so nothing constrains the sum of the outputs.
Example: for the input [2, −1, 0.5], ReLU gives [2, 0, 0.5], which sums to 2.5. Softmax, softmax(z)ᵢ = e^(zᵢ) / Σⱼ e^(zⱼ), turns the same vector into about [0.786, 0.039, 0.175], which sums to 1. ReLU adds cheap non-linearity in hidden layers; softmax turns the output layer's scores into class probabilities.
Typical follow-ups: why ReLU helped deep networks train, and what the dying ReLU problem is. Its gradient is exactly 1 for positive inputs, so it does not shrink gradients the way sigmoid and tanh do. But a unit whose input is always negative outputs 0, gets zero gradient and stops learning. Leaky ReLU, which keeps a small slope such as 0.01 for negative inputs, and GELU are common alternatives.