# Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/2606.12054 · HTML: https://mlsanity.com/paper/2606.12054/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [On the Generalization Benefit of Noise in Stochastic Gradient Descent](https://mlsanity.com/paper/2006.15081/) (2020)
- [Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss](https://mlsanity.com/paper/2205.13710/) (2022)
- [Anticorrelated Noise Injection for Improved Generalization](https://mlsanity.com/paper/2202.02831/) (2022)
- [Stochastic noise can be helpful for variational quantum algorithms](https://mlsanity.com/paper/2210.06723/) (2022)
- [Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD](https://mlsanity.com/paper/1906.10822/) (2019)
- [Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)](https://mlsanity.com/paper/2311.00521/) (2023)
- [Anisotropic Gaussian Smoothing for Gradient-based Optimization](https://mlsanity.com/paper/2411.11747/) (2024)
- [Perturbed Gradient Descent Algorithms are Small-Disturbance Input-to-State Stable](https://mlsanity.com/paper/2507.02131/) (2025)
- [Improved Performance of Stochastic Gradients with Gaussian Smoothing](https://mlsanity.com/paper/2311.00531/) (2023)
- [Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimization](https://mlsanity.com/paper/2405.01731/) (2024)

## Abstract (arXiv)

Injecting noise into the optimization process is a well-established technique for improving the training and generalization of deep neural networks. Yet, despite the breadth of existing approaches, it remains unclear which design choices truly matter in practice. In this work, we investigate parameter noise injection for stochastic gradient descent, focusing on two key questions: how to efficiently pair each training example with its own perturbation in mini-batch training, and whether sophisticated noise parameterizations or multi-sample gradient averaging yield meaningful gains over simpler alternatives. To address the first question, we leverage a distributional identity for linear layers that allows per-example noise injection without breaking batched computation. To address the second, we systematically compare several diagonal Gaussian parameterizations against an isotropic baseline across varying noise levels on CIFAR100. Our results consistently show that simple, lightweight strategies, isotropic noise with a single perturbed forward pass per update step, recover most of the benefit of more complex schemes. These findings suggest that simplicity suffices for parameter noise injection, and that practitioners need not resort to elaborate perturbation designs to reap the optimization and generalization benefits of noisy SGD.