Expected squared error splits into squared bias (a model too simple for the truth), variance (sensitivity to which training sample you drew) and irreducible noise. Pushing one down usually lifts the other, so you tune capacity and regularization to minimise the sum, not either term alone.
Card 1 of 259. Answer: Expected squared error splits into squared bias (a model too simple for the truth), variance (sensitivity to which training sample you drew) and irreducible noise. Pushing one down usually lifts the other, so you tune capacity and regularization to minimise the sum, not either term alone.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The bias-variance tradeoff is the tension between underfitting and overfitting. Bias is error from wrong assumptions, such as fitting a straight line to a curve. Variance is how much the fitted model changes when trained on a different sample. Simple models have high bias and low variance; flexible models the reverse.
For squared error the expected test error at a point decomposes exactly: E[(y − f̂(x))²] = Bias[f̂(x)]² + Var[f̂(x)] + σ², where σ² is irreducible noise. Fit polynomials to 20 noisy points: degree 1 underfits with high bias, degree 15 chases nearly every point and swings wildly between them, showing high variance. A middle degree minimizes the total.
To diagnose: high training and validation error points to bias, so add features or capacity. Low training error with much higher validation error points to variance, so add data, regularize or simplify. Bagging mainly cuts variance, boosting mainly cuts bias. Very large neural networks can show double descent, where test error falls again past the point of perfect training fit.
