Two sources of randomness: each tree is trained on a bootstrap sample of the rows, and each split considers only a random subset of the features, drawn afresh at every split. Averaging these decorrelated trees reduces variance.
Card 1 of 259. Answer: Two sources of randomness: each tree is trained on a bootstrap sample of the rows, and each split considers only a random subset of the features, drawn afresh at every split. Averaging these decorrelated trees reduces variance.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
"Random" refers to two sources of randomness. Each tree is trained on a bootstrap sample of the rows, drawn with replacement, and each split may only choose among a random subset of the features, drawn afresh at every split rather than once per tree. Without the feature sampling, the forest is just bagged trees.
Feature sampling matters when one feature dominates. Bagged trees would nearly all split on it at the root, so they would look alike and share mistakes. With 25 features and 5 candidates per split, the strong feature is available at a given split only 20% of the time, so the trees decorrelate, and averaging decorrelated trees cuts variance far more.
Tin Kam Ho's 1995 "random decision forests" grew each tree on one random feature subset; Leo Breiman's 2001 paper "Random Forests" combined bootstrap rows with per-split sampling, the version libraries implement. Defaults differ: scikit-learn's classifier tries √p features per split, but its regressor uses all p, which is bagging unless you lower max_features.
