Regularization: a penalty on parameter size added to the loss, so the fit stays smooth rather than chasing every training point. L1 (lasso) zeroes weights outright and selects features, L2 (ridge) shrinks them all. Its strength is tuned on validation data, never training loss.
Card 1 of 259. Answer: Regularization: a penalty on parameter size added to the loss, so the fit stays smooth rather than chasing every training point. L1 (lasso) zeroes weights outright and selects features, L2 (ridge) shrinks them all. Its strength is tuned on validation data, never training loss.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Regularization. It adds a penalty to the training loss, minimizing loss(w) + λ · penalty(w), where the tuning parameter λ sets how much complexity costs. Large weights let predictions swing sharply between nearby inputs, so penalizing them gives a smoother function that follows the trend, not every noisy point.
Smoothing splines show the idea literally. They minimize Σ (yᵢ − f(xᵢ))² + λ ∫ f''(x)² dx, where the integral measures total curvature. With λ = 0 the curve can pass through every training point; as λ grows, bending gets expensive, and in the limit the fit is the least-squares straight line. Ridge and lasso make the same trade on weights.
The follow-up is choosing λ. Training loss always prefers λ = 0, so pick it by cross-validation over a log-spaced grid, such as 0.001 to 100. In a broader sense, regularization is anything that limits effective complexity: dropout, early stopping, data augmentation or capping tree depth. Too much of it causes underfitting.
