# SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning

arXiv: https://arxiv.org/abs/1805.07898 · HTML: https://mlsanity.com/paper/1805.07898/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 25 of 25)

- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [Friendly Sharpness-Aware Minimization](https://mlsanity.com/paper/2403.12350/) (2024)
- [MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization](https://mlsanity.com/paper/2210.00102/) (2022)
- [How Well Does Self-Supervised Pre-Training Perform with Streaming Data?](https://mlsanity.com/paper/2104.12081/) (2021)
- [Scaling Law for Language Models Training Considering Batch Size](https://mlsanity.com/paper/2412.01505/) (2024)
- [Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape](https://mlsanity.com/paper/2409.14396/) (2024)
- [Activated Gradients for Deep Neural Networks](https://mlsanity.com/paper/2107.04228/) (2021)
- [Revisiting Random Weight Perturbation for Efficiently Improving Generalization](https://mlsanity.com/paper/2404.00357/) (2024)
- [DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing](https://mlsanity.com/paper/2103.01496/) (2021)
- [Efficient Generalization Improvement Guided by Random Weight Perturbation](https://mlsanity.com/paper/2211.11489/) (2022)
- [Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models](https://mlsanity.com/paper/2406.13137/) (2024)
- [Exploring Flat Minima for Domain Generalization with Large Learning Rates](https://mlsanity.com/paper/2309.06337/) (2023)
- [BN-invariant sharpness regularizes the training model to better generalization](https://mlsanity.com/paper/2101.02944/) (2021)
- [Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning](https://mlsanity.com/paper/2309.11305/) (2023)
- [Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?](https://mlsanity.com/paper/2408.04065/) (2024)
- [Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning](https://mlsanity.com/paper/2503.14205/) (2025)
- [Tackling benign nonconvexity with smoothing and stochastic gradients](https://mlsanity.com/paper/2202.09052/) (2022)
- [Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07372/) (2022)
- [Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks](https://mlsanity.com/paper/2502.12384/) (2025)
- [CurvFed: Curvature-Aligned Federated Learning for Fairness without Demographics](https://mlsanity.com/paper/2404.19725/) (2024)
- [Why is parameter averaging beneficial in SGD? An objective smoothing perspective](https://mlsanity.com/paper/2302.09376/) (2023)
- [Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method](https://mlsanity.com/paper/2309.14601/) (2023)
- [Random Aggregate Beamforming for Over-the-Air Federated Learning in Large-Scale Networks](https://mlsanity.com/paper/2403.18946/) (2024)
- [SMAT: Simple and Efficient Merge-Aware Training](https://mlsanity.com/paper/2609.33437/) (2026)

## Abstract (arXiv)

In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It becomes an open question whether escaping sharp minima can improve the generalization. To answer this question, we propose SmoothOut framework to smooth out sharp minima in DNNs and thereby improve generalization. In a nutshell, SmoothOut perturbs multiple copies of the DNN by noise injection and averages these copies. Injecting noises to SGD is widely used in the literature, but SmoothOut differs in lots of ways: (1) a de-noising process is applied before parameter updating; (2) noise strength is adapted to filter norm; (3) an alternative interpretation on the advantage of noise injection, from the perspective of sharpness and generalization; (4) usage of uniform noise instead of Gaussian noise. We prove that SmoothOut can eliminate sharp minima. Training multiple DNN copies is inefficient, we further propose an unbiased stochastic SmoothOut which only introduces the overhead of noise injecting and de-noising per batch. An adaptive variant of SmoothOut, AdaSmoothOut, is also proposed to improve generalization. In a variety of experiments, SmoothOut and AdaSmoothOut consistently improve generalization in both small-batch and large-batch training on the top of state-of-the-art solutions.