mlsanityData to 14 Oct 2026

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks

arXiv 2306.11680 ↗First posted 20 Jun 2023cs.LG math.OC stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an exp(-Omega(log2 t)) convergence rate. This distinguishes linear models with batch normalization from those without batch normalization in terms of both the type of implicit bias and the convergence rate. We further extend our result to a class of two-layer, single-filter linear convolutional neural networks, and show that batch normalization has an implicit bias towards a patch-wise uniform margin. Based on two examples, we demonstrate that patch-wise uniform margin classifiers can outperform the maximum margin classifiers in certain learning problems. Our results contribute to a better theoretical understanding of batch normalization.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

32 of 37 references.

References · 32

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Group Normalizationcited by 1,524
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  7. 2018How Does Batch Normalization Help Optimization?cited by 357
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  9. 2017Implicit Regularization in Matrix Factorizationcited by 322
  10. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  11. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
20 more
  1. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  2. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  3. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  4. 2018Understanding Batch Normalizationcited by 159
  5. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  6. 2018Convergence of Gradient Descent on Separable Datacited by 110
  7. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  8. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  9. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  10. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  11. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  12. 2019A Mean Field Theory of Batch Normalizationcited by 58
  13. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  14. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  15. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  16. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  17. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  18. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  19. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15
  20. 2023On the Training Instability of Shuffling SGD with Batch Normalizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 3.

8 papers.

  1. 2024The Implicit Bias of Adam on Separable Datacited by 36
  2. 2024On the Benefits of Over-parameterization for Out-of-Distribution Generalizationcited by 7
  3. 2024Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancycited by 2
  4. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  5. 2024Adversarial Vulnerability as a Consequence of On-Manifold Inseparibilitycited by 0
  6. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  7. 2026Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalizationcited by 0
  8. 2026A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Modelscited by 0