Machine Learning
Does feature selection tend to increase overfitting?
No. It actually could help to reduce overfitting.
Card 1 of 259. Answer: No. It actually could help to reduce overfitting.
No. It actually could help to reduce overfitting.
Card 1 of 259. Answer: No. It actually could help to reduce overfitting.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
No. Feature selection usually reduces overfitting. It keeps the features that carry useful signal and drops irrelevant or redundant ones, so the model has fewer parameters with which to fit noise. It also makes models faster and easier to explain.
Each extra feature lets a model fit a little more noise. Fit a linear regression on 100 rows with 50 pure-noise features: the expected training R² is 50/99, about 0.5, though the true R² is 0. Dropping such features removes that false fit and lowers variance. Common methods are filters such as mutual information, wrappers such as recursive feature elimination, and embedded methods such as L1 regularization.
The catch interviewers look for: selection overfits when it uses the data you evaluate on. The Elements of Statistical Learning describes 50 samples, 5,000 noise predictors and random labels. Picking the 100 predictors most correlated with the labels on all the data, then cross-validating a 1-nearest-neighbor classifier, gave about 3% error against a true 50%. Select features inside each fold.