The tension between underfitting and overfitting. Expected squared error splits into bias² (a model too simple for the truth), variance (sensitivity to the training sample) and irreducible noise. More complexity lowers bias and raises variance, so you tune it to minimize the sum.
Card 1 of 274. Answer: The tension between underfitting and overfitting. Expected squared error splits into bias² (a model too simple for the truth), variance (sensitivity to the training sample) and irreducible noise. More complexity lowers bias and raises variance, so you tune it to minimize the sum.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
The bias-variance tradeoff is the tension between underfitting and overfitting. Bias is error from wrong or overly rigid assumptions, such as fitting a straight line to a curve. Variance is how much the fitted model changes when it is trained on a different sample from the same population. Simple models have high bias and low variance; flexible models have low bias and high variance.
For squared error, the expected test error at a point decomposes exactly: E[(y − f̂(x))²] = Bias[f̂(x)]² + Var[f̂(x)] + σ², where σ² is irreducible noise. Fit polynomials to 20 noisy points: degree 1 underfits with high bias, while degree 15 chases nearly every point and swings wildly between them, showing high variance. A middle degree minimizes the total.
Model complexity pushes the two terms in opposite directions. Increasing complexity, with more parameters, deeper trees or higher-degree features, tends to decrease bias and increase variance, which leads to overfitting once variance dominates. Simplifying a model does the reverse: it increases bias and decreases variance, so a model simplified too far underfits because of the extra bias. In k-nearest neighbors, k = 1 copies the nearest training point (low bias, high variance), while k = 50 averages many points into smooth, stable, biased predictions. As complexity grows, test error usually falls and then climbs again: a U-shape.
A popular trick question says high bias is overfitting. It is underfitting. A high-bias model does poorly everywhere, say 70% training accuracy and 68% validation accuracy. A high-variance model memorizes the training data, noise included: 99% training accuracy but 75% on validation. Poor scores on both point to bias; a large gap between them points to variance.
The fixes follow from the diagnosis. For high bias, use a more flexible model, add better features, reduce regularization or train longer; more data rarely helps. For high variance, get more data, add regularization such as weight decay or dropout, stop early or simplify the model. To find the bottom of the U, cross-validate over a complexity knob such as tree depth, k or regularization strength. Bagging mainly cuts variance and boosting mainly cuts bias. One known exception to the U: very large neural networks can show double descent, where test error falls again after the model fits the training data perfectly.
