# Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learning

arXiv: https://arxiv.org/abs/2203.09962 · HTML: https://mlsanity.com/paper/2203.09962/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)
- [Deep Pyramidal Residual Networks](https://mlsanity.com/paper/1610.02915/) (2016)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning](https://mlsanity.com/paper/2202.03599/) (2022)
- [Sharpness-Aware Minimization Improves Language Model Generalization](https://mlsanity.com/paper/2110.08529/) (2021)
- [Regularizing Neural Networks via Adversarial Model Perturbation](https://mlsanity.com/paper/2010.04925/) (2020)
- [Randomized Gradient Boosting Machine](https://mlsanity.com/paper/1810.10158/) (2018)
- [Gradient boosting machine with partially randomized decision trees](https://mlsanity.com/paper/2006.11014/) (2020)

## Built on by (most cited first, 14 of 14)

- [A Universal Class of Sharpness-Aware Minimization Algorithms](https://mlsanity.com/paper/2406.03682/) (2024)
- [Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy](https://mlsanity.com/paper/2401.07250/) (2024)
- [SAMPa: Sharpness-aware Minimization Parallelized](https://mlsanity.com/paper/2410.10683/) (2024)
- [Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models](https://mlsanity.com/paper/2406.13137/) (2024)
- [Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization](https://mlsanity.com/paper/2406.08001/) (2024)
- [LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM](https://mlsanity.com/paper/2502.19571/) (2025)
- [Flat Minima and Generalization: Insights from Stochastic Convex Optimization](https://mlsanity.com/paper/2511.03548/) (2025)
- [When Will Gradient Regularization Be Harmful?](https://mlsanity.com/paper/2406.09723/) (2024)
- [Bilateral Sharpness-Aware Minimization for Flatter Minima](https://mlsanity.com/paper/2409.13173/) (2024)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius](https://mlsanity.com/paper/2408.08222/) (2024)
- [Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning](https://mlsanity.com/paper/2602.09395/) (2026)
- [Sharpness-aware Federated Graph Learning](https://mlsanity.com/paper/2512.16247/) (2025)
- [Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation](https://mlsanity.com/paper/2603.10048/) (2026)

## Abstract (arXiv)

By driving models to converge to flat minima, sharpness-aware learning algorithms (such as SAM) have shown the power to achieve state-of-the-art performances. However, these algorithms will generally incur one extra forward-backward propagation at each training iteration, which largely burdens the computation especially for scalable models. To this end, we propose a simple yet efficient training scheme, called Randomized Sharpness-Aware Training (RST). Optimizers in RST would perform a Bernoulli trial at each iteration to choose randomly from base algorithms (SGD) and sharpness-aware algorithms (SAM) with a probability arranged by a predefined scheduling function. Due to the mixture of base algorithms, the overall count of propagation pairs could be largely reduced. Also, we give theoretical analysis on the convergence of RST. Then, we empirically study the computation cost and effect of various types of scheduling functions, and give directions on setting appropriate scheduling functions. Further, we extend the RST to a general framework (G-RST), where we can adjust regularization degree on sharpness freely for any scheduling function. We show that G-RST can outperform SAM in most cases while saving 50% extra computation cost.