Machine Learning
Which regularization technique penalizes the sum of absolute value of weights? L1 or L2?
L1.
Card 1 of 259. Answer: L1.
L1.
Card 1 of 259. Answer: L1.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
L1. L1 regularization adds λ Σ |wᵢ|, λ times the L1 norm ‖w‖₁, to the training loss. The name comes from that norm, also called the Manhattan or taxicab norm: length measured as the sum of distances along each axis. L1-penalized linear regression is called lasso.
Example: weights (3, −2, 0.5) have an L1 norm of 5.5, so λ = 0.1 adds 0.55 to the loss; the sign is ignored. Because |w| has a kink at zero, the penalty has no gradient there. Solvers use coordinate descent, as scikit-learn's Lasso does, or proximal steps that shrink each weight by a fixed amount and set it to zero if it would cross. Plain SGD tends to leave weights jittering near zero.
A practical follow-up is scaling. The penalty charges raw weight size, so units matter: measure a distance in kilometers instead of meters and its weight must be 1,000 times larger, costing 1,000 times more, so lasso is far more likely to drop it. Standardize features first and leave the intercept unpenalized.