mlsanityData to 14 Oct 2026

Normalization Layers Are All That Sharpness-Aware Minimization Needs

arXiv 2306.04226 ↗First posted 7 Jun 2023cs.LG cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings. In this work we show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.This finding generalizes to different SAM variants and both ResNet (Batch Normalization) and Vision Transformer (Layer Normalization) architectures. We consider alternative sparse perturbation approaches and find that these do not achieve similar performance enhancement at such extreme sparsity levels, showing that this behaviour is unique to the normalization layers. Although our findings reaffirm the effectiveness of SAM in improving generalization performance, they cast doubt on whether this is solely caused by reduced sharpness.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 54 of its 61 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

54 of 61 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2016Layer Normalizationcited by 5,669
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2016Wide Residual Networkscited by 3,070
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  3. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  6. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  7. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  8. 2018How Does Batch Normalization Help Optimization?cited by 357
  9. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  10. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  11. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  12. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  13. 2019Understanding and Improving Layer Normalizationcited by 217
  14. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  15. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  16. 2018Understanding Batch Normalizationcited by 159
  17. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  18. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  19. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  20. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  21. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 104
  22. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 102
  23. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  24. 2020Relative Flatness and Generalizationcited by 88
  25. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  26. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  27. 2020In Search of Robust Measures of Generalizationcited by 79
  28. 2022When Do Flat Minima Optimizers Work?cited by 78
  29. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  30. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  31. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 58
  32. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  33. 2022SAM as an Optimal Relaxation of Bayescited by 41
  34. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  35. 2020Batch Normalization Increases Adversarial Vulnerability and Decreases Adversarial Transferability: A Non-Robust Feature Perspectivecited by 35
  36. 2019Four Things Everyone Should Know to Improve Batch Normalizationcited by 28
  37. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  38. 2021BN-NAS: Neural Architecture Search with Batch Normalizationcited by 21

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 11; 2026: 3.

27 papers.

Sort:

  1. 2025Transformers without Normalizationcited by 70
  2. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  3. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  4. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  5. 2023Layer-wise Linear Mode Connectivitycited by 22
  6. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 14
  7. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  8. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  9. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  10. 2025Stronger Normalization-Free Transformerscited by 13
  11. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  12. 2025Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentationcited by 7
15 more
  1. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  2. 2024Tilted Sharpness-Aware Minimizationcited by 6
  3. 2025Analyzing limits for in-context learningcited by 6
  4. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 3
  5. 2024Can Optimization Trajectories Explain Multi-Task Transfer?cited by 2
  6. 2025Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learningcited by 2
  7. 2025Do Flat Minima Improve Sparse Novel View Synthesis?cited by 1
  8. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0
  9. 2024Equivariant Test-Time Training with Operator Sketching for Imaging Inverse Problemscited by 0
  10. 2025Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Modelscited by 0
  11. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  12. 2025The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Controlcited by 0
  13. 2025Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapescited by 0
  14. 2026Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimizationcited by 0
  15. 2026SandwichQuant: Which Parameters Matter Before and After Quantization?cited by 0