Machine Learning
Which regularization technique penalizes the sum of square weights? L1 or L2?
L2.
Card 1 of 259. Answer: L2.
L2.
Card 1 of 259. Answer: L2.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
L2. L2 regularization adds λ Σ wᵢ², λ times the squared Euclidean norm ‖w‖₂², to the loss. It is the squared length of the weight vector, not the length itself, so it is smooth and differentiable everywhere. L2-penalized linear regression is called ridge; in neural networks it is usually called weight decay.
Its gradient is 2λw, so each gradient descent step multiplies every weight by (1 − 2ηλ) before the data gradient is applied. With learning rate η = 0.1 and λ = 0.01, weights shrink 0.2% per step unless the data pushes back. Texts that write (λ/2) Σ wᵢ² get the gradient λw; that only rescales λ.
Squaring makes large weights dominate: for weights (3, −2, 0.5) the penalty is 9 + 4 + 0.25 = 13.25, 68% of it from the 3. So L2 prefers to spread weight: if four near-identical features can share the work, one weight of 4 costs 16 and four weights of 1 cost 4, while L1 charges 4 either way. scikit-learn's LogisticRegression uses L2 by default.