The network can't learn useful features: every unit in a layer computes the same output and gets the same gradient, so the symmetry never breaks. With tanh or ReLU the weights don't move at all; with sigmoid they move in lockstep. Use small random weights instead.
Proposed by @Al_Grigor
Card 1 of 259. Answer: The network can't learn useful features: every unit in a layer computes the same output and gets the same gradient, so the symmetry never breaks. With tanh or ReLU the weights don't move at all; with sigmoid they move in lockstep. Use small random weights instead.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The network cannot learn useful features. With all weights at zero, every unit in a layer computes the same output and gets the same gradient, so the units never diverge. Depending on the activation, weights either never move or move in lockstep, so each layer acts like one neuron.
Take h = tanh(W₁x + b₁) and ŷ = W₂h + b₂, all at 0. Every hidden unit outputs tanh(0) = 0, so W₂'s gradient, the output error times h, is 0. The error sent back to the hidden layer is scaled by W₂ = 0, so W₁ gets none. Only b₂ learns, and the model predicts a constant. ReLU fails the same way. With sigmoid, h = 0.5, so W₂ changes, but all hidden units get identical updates.
The fix is small random weights, such as He or Xavier initialization; biases can start at zero. Zero can be deliberate: LoRA zeroes its B matrix so the adapter starts as a no-op, which works because its partner A is random.
