# Towards Understanding the Importance of Noise in Training Neural Networks

arXiv: https://arxiv.org/abs/1909.03172 · HTML: https://mlsanity.com/paper/1909.03172/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2019.

## Built on by (most cited first, 10 of 10)

- [PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation](https://mlsanity.com/paper/2202.00308/) (2022)
- [Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization](https://mlsanity.com/paper/2102.12430/) (2021)
- [Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis](https://mlsanity.com/paper/2104.04676/) (2021)
- [Learning Enriched Illuminants for Cross and Single Sensor Color Constancy](https://mlsanity.com/paper/2203.11068/) (2022)
- [Uncertainty Injection: A Deep Learning Method for Robust Optimization](https://mlsanity.com/paper/2302.12304/) (2023)
- [Perlin Noise Improve Adversarial Robustness](https://mlsanity.com/paper/2112.13408/) (2021)
- [Adaptive Heavy-Tailed Stochastic Gradient Descent](https://mlsanity.com/paper/2508.21353/) (2025)
- [Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provably](https://mlsanity.com/paper/2202.03535/) (2022)
- [Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithms](https://mlsanity.com/paper/2403.00574/) (2024)
- [Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes](https://mlsanity.com/paper/2511.17399/) (2025)

## Abstract (arXiv)

Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks. The theory behind, however, is still largely unknown. This paper studies this fundamental problem through training a simple two-layer convolutional neural network model. Although training such a network requires solving a nonconvex optimization problem with a spurious local optimum and a global optimum, we prove that perturbed gradient descent and perturbed mini-batch stochastic gradient algorithms in conjunction with noise annealing is guaranteed to converge to a global optimum in polynomial time with arbitrary initialization. This implies that the noise enables the algorithm to efficiently escape from the spurious local optimum. Numerical experiments are provided to support our theory.