Without a nonlinear activation, stacked layers collapse into one linear map: W₂(W₁x + b₁) + b₂ is just another linear function of x. Activation functions add the nonlinearity that lets a neural network learn complex functions, such as XOR, that no linear model can represent.
Proposed by @Al_Grigor
Card 1 of 276. Answer: Without a nonlinear activation, stacked layers collapse into one linear map: W₂(W₁x + b₁) + b₂ is just another linear function of x. Activation functions add the nonlinearity that lets a neural network learn complex functions, such as XOR, that no linear model can represent.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Without activation functions, a neural network of any depth can only represent a linear function of its input. An activation function applies a nonlinear function such as ReLU, tanh or sigmoid to each unit's weighted sum, and that nonlinearity lets stacked layers model curved decision boundaries and complex relationships.
The collapse is one line of algebra: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂), a single linear layer. Ten linear layers are still one. XOR shows what nonlinearity buys: no linear model outputs 0, 1, 1, 0 for the inputs (0,0), (0,1), (1,0), (1,1), but h₁ = ReLU(x₁ + x₂), h₂ = ReLU(x₁ + x₂ − 1) and y = h₁ − 2h₂ do exactly that.
Follow-ups: the universal approximation theorem says one hidden layer with any non-polynomial activation can approximate any continuous function on a bounded region, though it may need very many units. Regression output layers often use no activation. And a ReLU network is piecewise linear, not linear: each unit adds a bend.
