Bootstrap aggregating: train the same learner on many bootstrap resamples and average the predictions, which cuts variance and leaves bias roughly untouched. That is why it suits deep unstable trees, while boosting attacks bias by fitting each model to the last one's errors.
Card 1 of 259. Answer: Bootstrap aggregating: train the same learner on many bootstrap resamples and average the predictions, which cuts variance and leaves bias roughly untouched. That is why it suits deep unstable trees, while boosting attacks bias by fitting each model to the last one's errors.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Bagging stands for bootstrap aggregating. It trains many copies of one model, each on a bootstrap sample of the training data (n rows drawn with replacement from the n available), then aggregates their predictions: an average for regression, a majority vote for classification. Leo Breiman introduced it in 1994 and published "Bagging Predictors" in 1996.
Each bootstrap sample contains about 63.2% of the distinct rows, since a row is never drawn with probability (1 − 1/n)ⁿ ≈ 1/e ≈ 0.368. These out-of-bag rows give a free validation estimate. Averaging B models with variance σ² and pairwise correlation ρ gives variance ρσ² + (1 − ρ)σ²/B: more models help, but correlation sets a floor.
Bagging helps unstable, high-variance learners such as deep decision trees and does little for stable ones like linear regression. Random forests also sample features at each split to lower ρ. Boosting is the contrast: it trains models in sequence, each fixing the errors of the previous ones, and mainly reduces bias.
