The coefficients are no longer unique: z is a linear combination of x and y (perfect multicollinearity), so XᵀX is singular and infinitely many coefficient sets give the same fit. Predictions still work; drop z or use ridge regression.
Proposed by @Al_Grigor
Card 1 of 259. Answer: The coefficients are no longer unique: z is a linear combination of x and y (perfect multicollinearity), so XᵀX is singular and infinitely many coefficient sets give the same fit. Predictions still work; drop z or use ridge regression.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The coefficients stop being unique. Because z = x + y, the columns are linearly dependent, a case called perfect multicollinearity, so XᵀX in the least squares solution ŵ = (XᵀX)⁻¹Xᵀt, with t the target, is singular. Predictions are still well defined; only the coefficients cannot be pinned down.
Example: if t = x + 2y, the coefficients (1, 2, 0), (0, 1, 1) and (3, 4, −2) on (x, y, z) give identical predictions. Since x + y − z = 0, any multiple of (1, 1, −1) can be added without changing the fit, so 'the effect of z' has no answer.
R's lm reports NA for one coefficient, while NumPy's lstsq and scikit-learn return the minimum-norm solution, here (0, 1, 1); a literal matrix inverse may fail or return huge, meaningless numbers. Drop z or use ridge regression. Common follow-ups: the dummy variable trap, one-hot columns for every category plus an intercept, and z = x + y + small noise, which fits but gives unstable coefficients.
