# GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

arXiv: https://arxiv.org/abs/2210.06895 · HTML: https://mlsanity.com/paper/2210.06895/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle](https://mlsanity.com/paper/1905.00877/) (2019)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [TBT: Targeted Neural Network Attack with Bit Trojan](https://mlsanity.com/paper/1909.05193/) (2019)
- [Regularizing Neural Networks via Adversarial Model Perturbation](https://mlsanity.com/paper/2010.04925/) (2020)
- [Can Adversarial Weight Perturbations Inject Neural Backdoors?](https://mlsanity.com/paper/2008.01761/) (2020)
- [Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption](https://mlsanity.com/paper/2006.05620/) (2020)
- [LCA: Loss Change Allocation for Neural Network Training](https://mlsanity.com/paper/1909.01440/) (2019)
- [Gradient ℓ1 Regularization for Quantization Robustness](https://mlsanity.com/paper/2002.07520/) (2020)
- [Adversarial Parameter Defense by Multi-Step Risk Minimization](https://mlsanity.com/paper/2109.02889/) (2021)

## Built on by (most cited first, 13 of 13)

- [Enhancing Sharpness-Aware Optimization Through Variance Suppression](https://mlsanity.com/paper/2309.15639/) (2023)
- [Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimization](https://mlsanity.com/paper/2303.13087/) (2023)
- [FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning](https://mlsanity.com/paper/2310.16412/) (2023)
- [Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy](https://mlsanity.com/paper/2401.07250/) (2024)
- [Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards](https://mlsanity.com/paper/2602.18037/) (2026)
- [Exploring Flat Minima for Domain Generalization with Large Learning Rates](https://mlsanity.com/paper/2309.06337/) (2023)
- [Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Search](https://mlsanity.com/paper/2503.10404/) (2025)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Adaptive Adversarial Cross-Entropy Loss for Sharpness-Aware Minimization](https://mlsanity.com/paper/2406.14329/) (2024)
- [VASSO: Variance Suppression for Sharpness-Aware Minimization](https://mlsanity.com/paper/2509.02433/) (2025)
- [DGSAM: Domain Generalization via Individual Sharpness-Aware Minimization](https://mlsanity.com/paper/2503.23430/) (2025)
- [Sharpness-Aware Minimization with Z-Score Gradient Filtering](https://mlsanity.com/paper/2505.02369/) (2025)
- [EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations](https://mlsanity.com/paper/2608.15105/) (2026)

## Abstract (arXiv)

Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks. It demonstrates that flat minima tend to imply better generalization abilities. However, it has some difficulty implying SAM to some natural language tasks, especially to models with drastic gradient changes, such as RNNs. In this work, we analyze the relation between the flatness of the local minimum and its generalization ability from a novel and straightforward theoretical perspective. We propose that the shift of the training and test distributions can be equivalently seen as a virtual parameter corruption or perturbation, which can explain why flat minima that are robust against parameter corruptions or perturbations have better generalization performances. On its basis, we propose a Gradient-Strength based Adaptive Sharpness-Aware Minimization (GA-SAM) algorithm to help to learn algorithms find flat minima that generalize better. Results in various language benchmarks validate the effectiveness of the proposed GA-SAM algorithm on natural language tasks.