False: with zero weights every unit in a layer computes the same thing and receives the same gradient, so the symmetry never breaks and the layer behaves like a single neuron. Use small random values scaled to the layer's size: He (variance 2/fan_in) for ReLU, Xavier (variance 2/(fan_in + fan_out)) for tanh. Biases can start at zero.
Proposed by @svpino
Card 1 of 259. Answer: False: with zero weights every unit in a layer computes the same thing and receives the same gradient, so the symmetry never breaks and the layer behaves like a single neuron. Use small random values scaled to the layer's size: He (variance 2/fan_in) for ReLU, Xavier (variance 2/(fan_in + fan_out)) for tanh. Biases can start at zero.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. With zero weights every unit in a hidden layer computes the same output and receives the same update, so the units stay identical and the layer acts like one neuron. Weights must differ, at a scale that keeps signals stable across layers.
A unit with fan-in n sums n weighted inputs, so its variance grows with n · Var(w). He initialization uses variance 2/n for ReLU, the 2 offsetting the half of inputs that ReLU zeroes; for n = 512 the standard deviation is 0.0625. With a tiny 0.01 instead, each ReLU layer scales the signal by √(512 × 0.0001 / 2) = 0.16, leaving about 10⁻⁸ of it after 10 layers.
Xavier, or Glorot, initialization uses variance 2/(fan_in + fan_out), for tanh and sigmoid. Zero is fine where there is no symmetry to break: biases, and models without hidden layers, such as logistic regression. Some networks zero one piece on purpose, like the last batch-norm scale in each residual block, so the block starts as an identity map.
