mlsanityData to 7 Oct 2026

Sharpness-Aware Minimization Improves Language Model Generalization

arXiv 2110.08529 ↗First posted 16 Oct 2021cs.CL cs.LGcited by 108

Not on the map. Why?

Abstract (arXiv)

The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size. Comparatively little work has been done to improve the generalization of these models through better optimization. In this work, we show that Sharpness-Aware Minimization (SAM), a recently proposed optimization procedure that encourages convergence to flatter minima, can substantially improve the generalization of language models without much computational overhead. We show that SAM is able to boost performance on SuperGLUE, GLUE, Web Questions, Natural Questions, Trivia QA, and TyDiQA, with particularly large gains when training data for these tasks is limited.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
24 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2019When Does Label Smoothing Help?cited by 985
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  11. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  12. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  13. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  14. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  15. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  16. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  17. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  18. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  19. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  20. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  21. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  22. 2020Implicit Gradient Regularizationcited by 145
  23. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  24. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 20; 2023: 32; 2024: 25; 2025: 17; 2026: 13.

108 papers.

Sort:

By area:

  1. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  4. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  5. 2022When Do Flat Minima Optimizers Work?cited by 78
  6. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  7. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  8. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  9. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  10. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  11. 2022SAM as an Optimal Relaxation of Bayescited by 40
  12. 2024Elucidating the Design Space of Dataset Condensationcited by 40
38 more
  1. 2022DSI++: Updating Transformer Memory with New Documentscited by 35
  2. 2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
  3. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  4. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  5. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  6. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  7. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  8. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  9. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  10. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  11. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  12. 2023Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Termcited by 16
  13. 2022Towards Bridging the Performance Gaps of Joint Energy-based Modelscited by 15
  14. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  15. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  16. 2022Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learningcited by 14
  17. 2023Sharpness-Aware Minimization Alone can Improve Adversarial Robustnesscited by 14
  18. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  19. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  20. 2023How to escape sharp minima with random perturbationscited by 13
  21. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  22. 2025Understanding Silent Data Corruption in LLM Trainingcited by 13
  23. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  24. 2022K-SAM: Sharpness-Aware Minimization at the Speed of SGDcited by 10
  25. 2023Flat Seeking Bayesian Neural Networkscited by 10
  26. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  27. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  28. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  29. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averagingcited by 9
  30. 2023Multi-Dataset Co-Training with Sharpness-Aware Optimization for Audio Anti-spoofingcited by 9
  31. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  32. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  33. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  34. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  35. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  36. 2024Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Modelscited by 7
  37. 2021Sharpness-Aware Minimization with Dynamic Reweightingcited by 6
  38. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6

50 most cited of 108 All 108 (JSON)