mlsanityData to 14 Oct 2026

Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization

arXiv 2303.03108 ↗First posted 3 Mar 2023cs.LG cs.CVcited by 57

Not on the map. Why?

Abstract (arXiv)

Recently, flat minima are proven to be effective for improving generalization and sharpness-aware minimization (SAM) achieves state-of-the-art performance. Yet the current definition of flatness discussed in SAM and its follow-ups are limited to the zeroth-order flatness (i.e., the worst-case loss within a perturbation radius). We show that the zeroth-order flatness can be insufficient to discriminate minima with low generalization error from those with high generalization error both when there is a single minimum or multiple minima within the given perturbation radius. Thus we present first-order flatness, a stronger measure of flatness focusing on the maximal gradient norm within a perturbation radius which bounds both the maximal eigenvalue of Hessian at local minima and the regularization function of SAM. We also present a novel training procedure named Gradient norm Aware Minimization (GAM) to seek minima with uniformly small curvature across all directions. Experimental results show that GAM improves the generalization of models trained with current optimizers such as SGD and AdamW on various datasets and networks. Furthermore, we show that GAM can help SAM find flatter minima and achieve better generalization.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 68 of its 98 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

68 of 98 references.

References · 68

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2016Wide Residual Networkscited by 3,070
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  10. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  12. 2019On the Convergence of Adam and Beyondcited by 841
  13. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  14. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  15. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  16. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  17. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  18. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  19. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  20. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 328
  21. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  22. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  23. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  24. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  25. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  26. 2017Three Factors Influencing Minima in SGDcited by 273
  27. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  28. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  29. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  30. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  31. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  32. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  33. 2016Deep Pyramidal Residual Networkscited by 163
  34. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  35. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  36. 2020Implicit Gradient Regularizationcited by 146
  37. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  38. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 21; 2025: 13; 2026: 15.

57 papers.

Sort:

By area:

  1. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  2. 2024Make Continual Learning Stronger via C-Flatcited by 35
  3. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  4. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  5. 2024Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learningcited by 26
  6. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 21
  7. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  8. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  9. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  10. 2024Domain-Inspired Sharpness-Aware Minimization Under Domain Shiftscited by 12
  11. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  12. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
38 more
  1. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  2. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  3. 2024Unknown Domain Inconsistency Minimization for Domain Generalizationcited by 7
  4. 2024Boosting Adversarial Transferability with Low-Cost Optimization via Maximin Expected Flatnesscited by 7
  5. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  6. 2023Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatnesscited by 5
  7. 2024DPO: Dual-Perturbation Optimization for Test-time Adaptation in 3D Object Detectioncited by 5
  8. 2024SZU-AFS Antispoofing System for the ASVspoof 5 Challengecited by 5
  9. 2024Optimal Parameter and Neuron Pruning for Out-of-Distribution Detectioncited by 4
  10. 2024Flatness-aware Sequential Learning Generates Resilient Backdoorscited by 3
  11. 2025Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learningcited by 3
  12. 2024CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalizationcited by 2
  13. 2024Bilateral Sharpness-Aware Minimization for Flatter Minimacited by 2
  14. 2024GAQAT: gradient-adaptive quantization-aware training for domain generalizationcited by 2
  15. 2025Quantization Meets OOD: Generalizable Quantization-aware Training from a Flatness Perspectivecited by 2
  16. 2025Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learningcited by 2
  17. 2026OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Frameworkcited by 2
  18. 2023Signal Processing Meets SGD: From Momentum to Filtercited by 1
  19. 2023Continual Learning through Networks Splitting and Merging with Dreaming-Meta-Weighted Model Fusioncited by 1
  20. 2024Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimizationcited by 1
  21. 2024Deep Companion Learning: Enhancing Generalization Through Historical Consistencycited by 1
  22. 2024Meta Curvature-Aware Minimization for Domain Generalizationcited by 1
  23. 2025Exploring Scale Shift in Crowd Localization under the Context of Domain Generalizationcited by 1
  24. 2025ODP-Bench: Benchmarking Out-of-Distribution Performance Predictioncited by 1
  25. 2026Dynamic Momentum Recalibration in Online Gradient Learningcited by 1
  26. 2026A Faster Path to Continual Learningcited by 1
  27. 2026Estimating Implicit Regularization in Deep Learningcited by 1
  28. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
  29. 2026One-Step Generalization Ratio Guided Optimization for Domain Generalizationcited by 1
  30. 2024Better YOLO with Attention-Augmented Network and Enhanced Generalization Performance for Safety Helmet Detectioncited by 0
  31. 2025DGSAM: Domain Generalization via Individual Sharpness-Aware Minimizationcited by 0
  32. 2025Adaptively Sampling-Reusing-Mixing Decomposed Gradients to Speed Up Sharpness Aware Minimizationcited by 0
  33. 2025Learning Regularizers: Learning Optimizers that can Regularizecited by 0
  34. 2025Beyond Deceptive Flatness: Dual-Order Solution for Strengthening Adversarial Transferabilitycited by 0
  35. 2025Learning to Generate Cross-Task Unexploitable Examplescited by 0
  36. 2025Sharpness-aware Federated Graph Learningcited by 0
  37. 2026Beyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learningcited by 0
  38. 2026Rethinking Graph Generalization through the Lens of Sharpness-Aware Minimizationcited by 0

50 most cited of 57 All 57 (JSON)