mlsanityData to 14 Oct 2026

Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning

arXiv 2503.11147 ↗First posted 14 Mar 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) is an optimization method that improves the generalization performance of machine learning models. Despite its effectiveness, it has not been widely adopted in real-world applications due to its high computational cost. Existing approaches reduce this overhead by approximating or reusing gradients in the perturbation step, but still assume sequential perturbation and update steps. In this study, we introduce AsyncSAM, a variant of SAM that enables concurrent model perturbation and parameter updates across multiple compute resources. By adjusting the perturbation batch size in a system-aware manner, our asynchronous scheme fully overlaps perturbation with updates, eliminating the overhead and allowing even CPUs to perform perturbations without extra time. We also provide theoretical analysis showing that AsyncSAM achieves the same order of convergence as conventional SAM. Extensive experiments demonstrate that AsyncSAM delivers superior generalization performance and the highest training throughput among state-of-the-art SAM variants. For example, in ViT fine-tuning on CIFAR-100, it achieves accuracy comparable to SAM while maintaining training time nearly identical to SGD.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

23 of 34 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Wide Residual Networkscited by 3,070
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  7. 2019SGD: General Analysis and Improved Ratescited by 207
  8. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  9. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  10. 2017Revisiting Distributed Synchronous SGDcited by 142
  11. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  12. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
11 more
  1. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  2. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
  3. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  4. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  5. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 37
  6. 2021Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphscited by 33
  7. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  8. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  9. 2021DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systemscited by 10
  10. 2018Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learningcited by 9
  11. 2024Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Trainingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  2. 2025LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimizationcited by 0