mlsanityData to 7 Oct 2026

GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

arXiv 2210.06895 ↗First posted 13 Oct 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks. It demonstrates that flat minima tend to imply better generalization abilities. However, it has some difficulty implying SAM to some natural language tasks, especially to models with drastic gradient changes, such as RNNs. In this work, we analyze the relation between the flatness of the local minimum and its generalization ability from a novel and straightforward theoretical perspective. We propose that the shift of the training and test distributions can be equivalently seen as a virtual parameter corruption or perturbation, which can explain why flat minima that are robust against parameter corruptions or perturbations have better generalization performances. On its basis, we propose a Gradient-Strength based Adaptive Sharpness-Aware Minimization (GA-SAM) algorithm to help to learn algorithms find flat minima that generalize better. Results in various language benchmarks validate the effectiveness of the proposed GA-SAM algorithm on natural language tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

27 of 45 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  6. 2016Adversarial examples in the physical worldcited by 1,762
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 462
15 more
  1. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  2. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  3. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  4. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  5. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  6. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 181
  7. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  8. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  9. 2019TBT: Targeted Neural Network Attack with Bit Trojancited by 112
  10. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  11. 2020Can Adversarial Weight Perturbations Inject Neural Backdoors?cited by 51
  12. 2020Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruptioncited by 26
  13. 2019LCA: Loss Change Allocation for Neural Network Trainingcited by 9
  14. 2020Gradient ℓ1 Regularization for Quantization Robustnesscited by 4
  15. 2021Adversarial Parameter Defense by Multi-Step Risk Minimizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 5; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  2. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29
  3. 2023FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learningcited by 20
  4. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  5. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  6. 2023Exploring Flat Minima for Domain Generalization with Large Learning Ratescited by 4
  7. 2025Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Searchcited by 2
  8. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  9. 2024Adaptive Adversarial Cross-Entropy Loss for Sharpness-Aware Minimizationcited by 1
  10. 2025VASSO: Variance Suppression for Sharpness-Aware Minimizationcited by 1
  11. 2025DGSAM: Domain Generalization via Individual Sharpness-Aware Minimizationcited by 0
  12. 2025Sharpness-Aware Minimization with Z-Score Gradient Filteringcited by 0
1 more
  1. 2026EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbationscited by 0