Bootstrap aggregating: train the same model on many bootstrap samples of the data, drawn with replacement, and average or vote their predictions. It mainly cuts variance, which is why it suits deep, unstable trees.
Card 1 of 277. Answer: Bootstrap aggregating: train the same model on many bootstrap samples of the data, drawn with replacement, and average or vote their predictions. It mainly cuts variance, which is why it suits deep, unstable trees.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Bagging, short for bootstrap aggregating, trains many copies of one model, each on its own bootstrap sample of the training data, then aggregates their predictions: an average for regression, a majority vote for classification. A bootstrap sample draws n rows with replacement from the n available. Leo Breiman introduced the method in a 1994 technical report and published "Bagging Predictors" in 1996.
Bagging does not split the data into separate bags, despite a popular true-or-false question that says so. Because sampling is with replacement, the bags overlap and repeat rows. The chance that a given row is never drawn in n draws is (1 − 1/n)ⁿ, which approaches 1/e ≈ 0.368, so each bag holds about 63.2% of the distinct rows. The rows a model never saw, its out-of-bag rows, give a free validation estimate. Sampling without replacement instead is called pasting.
Averaging is what cuts the variance. For B models that each have variance σ² and pairwise correlation ρ, the average has variance ρσ² + (1 − ρ)σ²/B. With σ² = 1 and ρ = 0.3, 10 models give 0.3 + 0.07 = 0.37 and 100 models give 0.307: more models help, but the correlation sets a floor that no number of models removes.
That is why bagging helps unstable, high-variance learners such as deep decision trees and does little for stable ones like linear regression. A random forest is bagged trees plus a random subset of features at each split, which lowers ρ. The models train in parallel, and adding more of them does not cause overfitting.
Interviewers usually follow up with boosting, the other main ensemble family. Boosting trains models one after another, each focusing on the examples the ensemble so far gets wrong, and mainly reduces bias rather than variance. So bagging wants strong, unstable base models, while boosting wants weak ones such as one-split stumps.
