mlsanityData to 14 Oct 2026

Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction

arXiv 2206.07085 ↗First posted 14 Jun 2022cs.LG cs.AIcited by 86

Not on the map. Why?

Abstract (arXiv)

Normalization layers (e.g., Batch Normalization, Layer Normalization) were introduced to help with optimization difficulties in very deep nets, but they clearly also help generalization, even in not-so-deep nets. Motivated by the long-held belief that flatter minima lead to better generalization, this paper gives mathematical analysis and supporting experiments suggesting that normalization (together with accompanying weight-decay) encourages GD to reduce the sharpness of loss surface. Here "sharpness" is carefully defined given that the loss is scale-invariant, a known consequence of normalization. Specifically, for a fairly broad class of neural nets with normalization, our theory explains how GD with a finite learning rate enters the so-called Edge of Stability (EoS) regime, and characterizes the trajectory of GD in this regime via a continuous sharpness-reduction flow.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 99 of its 129 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

99 of 129 references.

References · 99

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Layer Normalizationcited by 5,669
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  12. 2018Group Normalizationcited by 1,524
38 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2017Exploring Generalization in Deep Learningcited by 600
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  10. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  11. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  13. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  14. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  15. 2018How Does Batch Normalization Help Optimization?cited by 357
  16. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  17. 2017Implicit Regularization in Matrix Factorizationcited by 322
  18. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  19. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  20. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  21. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  22. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  23. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  24. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  25. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  26. 2020Directional convergence and alignment in deep learningcited by 178
  27. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 160
  28. 2018Understanding Batch Normalizationcited by 159
  29. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  30. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  31. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  32. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  33. 2020Implicit Gradient Regularizationcited by 146
  34. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  35. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  36. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  37. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  38. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 24; 2024: 18; 2025: 19; 2026: 17.

86 papers.

Sort:

By area:

  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  2. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  4. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  5. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  6. 2025Transformers without Normalizationcited by 70
  7. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  8. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  9. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  10. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  11. 2022Learning threshold neurons via the "edge of stability"cited by 51
  12. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
38 more
  1. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  2. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  3. 2024Does SGD really happen in tiny subspaces?cited by 36
  4. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  5. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  6. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  7. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
  8. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  9. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  10. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  11. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  12. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  13. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 18
  14. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 17
  15. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  16. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  17. 2023Loss Spike in Training Neural Networkscited by 16
  18. 2023Latent State Models of Training Dynamicscited by 16
  19. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  20. 2023Optimal Sample Complexity of Contrastive Learningcited by 15
  21. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  22. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  23. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  24. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  25. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  26. 2025Stronger Normalization-Free Transformerscited by 13
  27. 2024Towards Robust Out-of-Distribution Generalization Bounds via Sharpnesscited by 11
  28. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  29. 2024On the Nonlinearity of Layer Normalizationcited by 9
  30. 2023On progressive sharpening, flat minima and generalisationcited by 7
  31. 2023Exploring Weight Balancing on Long-Tailed Recognition Problemcited by 7
  32. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 7
  33. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  34. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  35. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  36. 2026The Origin of Edge of Stabilitycited by 6
  37. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  38. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4

50 most cited of 86 All 86 (JSON)