False — it is a normalization layer, not a backpropagation technique. It standardises each activation across the mini-batch, then rescales with learned parameters, which smooths optimisation and allows higher learning rates. Batch statistics also add noise, so it regularises mildly.
Proposed by @svpino
Card 1 of 259. Answer: False — it is a normalization layer, not a backpropagation technique. It standardises each activation across the mini-batch, then rescales with learned parameters, which smooths optimisation and allows higher learning rates. Batch statistics also add noise, so it regularises mildly.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
False. Batch normalization is a layer, not a backpropagation technique: backpropagation computes gradients through it like any other layer. It standardizes each feature with the mini-batch mean and variance, then applies a learned scale γ and shift β. Sergey Ioffe and Christian Szegedy introduced it in 2015.
Example: a unit outputs 2, 4, 6 and 8 across a batch of four. Mean and variance are both 5, so x̂ = (x − μ)/√(σ² + ε) gives about −1.34, −0.45, 0.45 and 1.34; with γ = 2 and β = 1, y = γx̂ + β gives −1.68, 0.11, 1.89 and 3.68. Learned γ and β can undo this if useful, and batch noise adds mild regularization.
At inference the layer instead uses running averages of mean and variance collected during training. Small batches make the statistics noisy, one reason transformers use layer normalization. The paper credited reduced internal covariate shift; Santurkar et al. (2018) argued the benefit is a smoother loss landscape, which allows higher learning rates.
