mlsanityData to 14 Oct 2026

Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization

arXiv 2501.12666 ↗First posted 22 Jan 2025cs.LG cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has attracted significant attention for its effectiveness in improving generalization across various tasks. However, its underlying principles remain poorly understood. In this work, we analyze SAM's training dynamics using the maximum eigenvalue of the Hessian as a measure of sharpness, and propose a third-order stochastic differential equation (SDE), which reveals that the dynamics are driven by a complex mixture of second- and third-order terms. We show that alignment between the perturbation vector and the top eigenvector is crucial for SAM's effectiveness in regularizing sharpness, but find that this alignment is often inadequate in practice, limiting SAM's efficiency. Building on these insights, we introduce Eigen-SAM, an algorithm that explicitly aims to regularize the top Hessian eigenvalue by aligning the perturbation vector with the leading eigenvector. We validate the effectiveness of our theory and the practical advantages of our proposed approach through comprehensive experiments. Code is available at https://github.com/RitianLuo/EigenSAM.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

32 of 48 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2016Wide Residual Networkscited by 3,070
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  10. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  11. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
20 more
  1. 2020Implicit Gradient Regularizationcited by 146
  2. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  3. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  4. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  5. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  6. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  7. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  8. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  9. 2022When Do Flat Minima Optimizers Work?cited by 78
  10. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  11. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  12. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  13. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  14. 2017Stochastic Methods for Composite and Weakly Convex Optimization Problemscited by 48
  15. 2024Does SGD really happen in tiny subspaces?cited by 36
  16. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  17. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  18. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  19. 2023How to escape sharp minima with random perturbationscited by 13
  20. 2024Leakage-Resilient and Carbon-Neutral Aggregation Featuring the Federated AI-enabled Critical Infrastructurecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  2. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  3. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  4. 2025DisarmRAG: Stealthy Retriever-Centric Poisoning to Disable Self-Correction in Retrieval-Augmented Generation (Extended Version)cited by 2
  5. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  6. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  7. 2025Do Flat Minima Improve Sparse Novel View Synthesis?cited by 1
  8. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  9. 2026Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networkscited by 1
  10. 2026Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimizationcited by 1
  11. 2025DGSAM: Domain Generalization via Individual Sharpness-Aware Minimizationcited by 0
  12. 2025Dynamic Spectral Backpropagation for Efficient Neural Network Trainingcited by 0
9 more
  1. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  2. 2025Eigen-Value: Efficient Domain-Robust Data Valuation via Eigenvalue-Based Approachcited by 0
  3. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  4. 2026Towards Compact and Robust DNNs via Compression-aware Sharpness Minimizationcited by 0
  5. 2026Decentralized SGD with Controlled Disagreement Finds Flatter Minimacited by 0
  6. 2026Curvature-Aligned Probing for Local Loss-Landscape Stabilizationcited by 0
  7. 2026Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Datacited by 0
  8. 2026Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networkscited by 0
  9. 2026On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Boundscited by 0