# Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD

arXiv: https://arxiv.org/abs/1906.10822 · HTML: https://mlsanity.com/paper/1906.10822/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2019.

## Built on by (most cited first, 7 of 7)

- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy](https://mlsanity.com/paper/2401.07250/) (2024)
- [Tackling benign nonconvexity with smoothing and stochastic gradients](https://mlsanity.com/paper/2202.09052/) (2022)
- [On the Complexity-Faithfulness Trade-off of Gradient-Based Explanations](https://mlsanity.com/paper/2508.10490/) (2025)
- [Why is parameter averaging beneficial in SGD? An objective smoothing perspective](https://mlsanity.com/paper/2302.09376/) (2023)
- [Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descent](https://mlsanity.com/paper/2606.12054/) (2026)

## Abstract (arXiv)

Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep learning because of its training-time efficiency, however, extremely large-batch SGD leads to poor generalization and easily converges to sharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from converging to good minima. To overcome this difficulty, we propose gradient noise convolution (GNC), which effectively smooths sharper minima of the loss function. For DP-SGD, GNC utilizes so-called gradient noise, which is induced by stochastic gradient variation and convolved to the loss function as a smoothing effect. GNC computation can be performed by simply computing the stochastic gradient on each parallel worker and merging them, and is therefore extremely easy to implement. Due to convolving with the gradient noise, which tends to spread along a sharper direction of the loss function, GNC can effectively smooth sharp minima and achieve better generalization, whereas isotropic random noise cannot. We empirically show this effect by comparing GNC with isotropic random noise, and show that it achieves state-of-the-art generalization performance for large-scale deep neural network optimization.