Machine Learning
Is it possible to design a linear regression algorithm using a neural network?
Yes.
Card 1 of 259. Answer: Yes.
Yes.
Card 1 of 259. Answer: Yes.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Yes. A network with no hidden layer, one output unit, an identity activation and mean squared error loss is exactly linear regression: ŷ = w·x + b, with the weights as coefficients and the bias as the intercept. Only the fitting method differs.
Ordinary least squares solves for the weights in one step with the normal equation, w = (XᵀX)⁻¹Xᵀy; the network finds them by gradient descent. On the points (1, 3), (2, 5) and (3, 7), both routes reach w = 2 and b = 1: the loss is convex with one minimum, which gradient descent finds with a suitable learning rate. Gradient descent pays off when the data is too large for XᵀX or arrives as a stream.
Follow-ups: stacking linear layers adds nothing, because W₂(W₁x) = (W₂W₁)x is still one linear map; non-linear activations are what make a network more than a regression. Swap in a sigmoid and log loss and the same neuron is logistic regression; add an L2 penalty and it is ridge regression.