The output of the sigmoid function for large positive or negative numbers is almost zero. From this comes the problem of vanishing gradient.
Proposed by @Al_Grigor
Card 1 of 259. Answer: The output of the sigmoid function for large positive or negative numbers is almost zero. From this comes the problem of vanishing gradient.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Sigmoid has three problems as a hidden-layer activation: it saturates, causing vanishing gradients; its output is not zero-centered; and it is relatively expensive to compute.
Saturation is the big one. For large positive or negative inputs the curve is flat, so its derivative is close to zero, and even at its peak the derivative is only 0.25. Backpropagation multiplies these derivatives layer after layer, so in a deep network the gradient reaching the early layers shrinks toward zero and those layers barely learn.
Because sigmoid outputs are always positive, the gradients for a neuron's weights all share the same sign, which pushes updates into an inefficient zig-zag. That is why ReLU and its variants replaced sigmoid in hidden layers, and tanh is preferred when a bounded output is needed. Sigmoid is still the right choice in the output layer for binary classification and inside the gates of LSTMs and GRUs.
