mlsanityData to 14 Oct 2026

Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems

arXiv 2410.14802 ↗First posted 18 Oct 2024cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization of SAM for scale-invariant problems involving two groups of variables. Instead of focusing on commonly used sharpness, this work introduces a concept termed balancedness, defined as the difference between the squared norm of two variables. This allows us to depict richer global behaviors of SAM. In particular, our theoretical and empirical findings reveal that i) SAM promotes balancedness; and ii) the regularization on balancedness is data-responsive -- outliers have stronger impact. The latter coincides with empirical observations that SAM outperforms SGD in the presence of outliers. Leveraging the implicit regularization, we develop a resource-efficient SAM variant, balancedness-aware regularization (BAR), tailored for scale-invariant problems such as finetuning language models with LoRA. BAR saves 95% computational overhead of SAM, with enhanced test performance across various tasks on RoBERTa, GPT2, and OPT-1.3B.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 74 of its 90 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

74 of 90 references.

References · 74

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
38 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2017Exploring Generalization in Deep Learningcited by 600
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  13. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  14. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  15. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  16. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  17. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  18. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  19. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  20. 2017Three Factors Influencing Minima in SGDcited by 273
  21. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  22. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  23. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  24. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  25. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  26. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  27. 2020Implicit Gradient Regularizationcited by 146
  28. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  29. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  30. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 129
  31. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  32. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  33. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  34. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  35. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  36. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  37. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  38. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 2.

10 papers.

  1. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  2. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  3. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 10
  4. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  5. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  6. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 5
  7. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  8. 2026Low-Rank Adaptation Redux for Large Modelscited by 3
  9. 2025Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithmcited by 1
  10. 2026The Sharp Directions Are Against You: Curvature Analysis of Activation Steeringcited by 0