mlsanityData to 7 Oct 2026

K-SAM: Sharpness-Aware Minimization at the Speed of SGD

arXiv 2210.12864 ↗First posted 23 Oct 2022cs.LG cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks. However, SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. The computational challenge posed by SAM arises because each iteration requires both ascent and descent steps and thus double the gradient computations. To address this challenge, we propose to compute gradients in both stages of SAM on only the top-k samples with highest loss. K-SAM is simple and extremely easy-to-implement while providing significant generalization boosts over vanilla SGD at little to no additional cost.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 21 of its 21 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

21 of 21 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  11. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
9 more
  1. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  2. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  3. 2016Deep Pyramidal Residual Networkscited by 163
  4. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  5. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  6. 2018A Walk with SGDcited by 74
  7. 2019Understanding Generalization through Visualizationscited by 54
  8. 2017Learning with Average Top-k Losscited by 52
  9. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 3; 2026: 1.

10 papers.

  1. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  2. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  3. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5
  4. 2024Improving SAM Requires Rethinking its Optimization Formulationcited by 4
  5. 2024Bilateral Sharpness-Aware Minimization for Flatter Minimacited by 2
  6. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  7. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 1
  8. 2025Domain-Generalization to Improve Learning in Meta-Learning Algorithmscited by 0
  9. 2025Sharpness-aware Federated Graph Learningcited by 0
  10. 2026Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimizationcited by 0