Deep Learning
Softmax is an activation function that always returns the input value if it’s positive and zeroes otherwise.
False
Proposed by @svpino
Card 1 of 259. Answer: False
False
Proposed by @svpino
Card 1 of 259. Answer: False
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. That describes ReLU, max(0, x). Softmax turns a whole vector of scores into a probability distribution: softmax(z)ᵢ = e^(zᵢ) / Σⱼ e^(zⱼ). Every output depends on every input and lies strictly between 0 and 1, and the outputs sum to 1, so softmax neither passes positive values through nor zeroes out negative ones.
Example: for the scores [3, 0, −3], ReLU returns [3, 0, 0]. Softmax exponentiates them to about 20.09, 1 and 0.05, which sum to 21.14, and returns about [0.950, 0.047, 0.002]. The 3 becomes a probability, and the negative score still gets a small positive share.
The two also live in different places: ReLU acts element by element in hidden layers, softmax across the output layer of a multi-class classifier. Their gradients show it: ReLU's derivative is 1 or 0 per unit, while softmax's, ∂pᵢ/∂zⱼ = pᵢ(δᵢⱼ − pⱼ), couples all outputs. The name misleads too: softmax smooths argmax, not max, which is why Goodfellow, Bengio and Courville suggest calling it softargmax.