# On the Inherent Regularization Effects of Noise Injection During Training

arXiv: https://arxiv.org/abs/2102.07379 · HTML: https://mlsanity.com/paper/2102.07379/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Adversarial Logit Pairing](https://mlsanity.com/paper/1803.06373/) (2018)
- [Generalized Sliced Wasserstein Distances](https://mlsanity.com/paper/1902.00434/) (2019)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Generalisation error in learning with random features and the hidden manifold model](https://mlsanity.com/paper/2002.09339/) (2020)
- [Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack](https://mlsanity.com/paper/1811.09310/) (2018)
- [The Gaussian equivalence of generative models for learning with shallow neural networks](https://mlsanity.com/paper/2006.14709/) (2020)
- [The Implicit and Explicit Regularization Effects of Dropout](https://mlsanity.com/paper/2002.12915/) (2020)
- [The role of regularization in classification of high-dimensional noisy Gaussian mixture](https://mlsanity.com/paper/2002.11544/) (2020)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [Efficient Defenses Against Adversarial Attacks](https://mlsanity.com/paper/1707.06728/) (2017)
- [Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization](https://mlsanity.com/paper/1805.10939/) (2018)
- [A framework to characterize performance of LASSO algorithms](https://mlsanity.com/paper/1303.7291/) (2013)
- [The Impact of Regularization on High-dimensional Logistic Regression](https://mlsanity.com/paper/1906.03761/) (2019)
- [MaxUp: A Simple Way to Improve Generalization of Neural Network Training](https://mlsanity.com/paper/2002.09024/) (2020)
- [Analyzing noise in autoencoders and deep networks](https://mlsanity.com/paper/1406.1831/) (2014)
- [The Implicit Regularization of Ordinary Least Squares Ensembles](https://mlsanity.com/paper/1910.04743/) (2019)
- [Phase Transitions in Transfer Learning for High-Dimensional Perceptrons](https://mlsanity.com/paper/2101.01918/) (2021)
- [Universality laws for randomized dimension reduction, with applications](https://mlsanity.com/paper/1511.09433/) (2015)
- [Symbol Error Rate Performance of Box-relaxation Decoders in Massive MIMO](https://mlsanity.com/paper/1711.11215/) (2017)

## Built on by (most cited first, 18 of 18)

- [Precise Asymptotic Analysis of Deep Random Feature Models](https://mlsanity.com/paper/2302.06210/) (2023)
- [On Regularization via Early Stopping for Least Squares Regression](https://mlsanity.com/paper/2406.04425/) (2024)
- [On how to avoid exacerbating spurious correlations when models are overparameterized](https://mlsanity.com/paper/2206.12739/) (2022)
- [Least Squares Regression Can Exhibit Under-Parameterized Double Descent](https://mlsanity.com/paper/2305.14689/) (2023)
- [A Novel Gaussian Min-Max Theorem and its Applications](https://mlsanity.com/paper/2402.07356/) (2024)
- [Noise Injection as a Probe of Deep Learning Dynamics](https://mlsanity.com/paper/2210.13599/) (2022)
- [Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance](https://mlsanity.com/paper/2402.11742/) (2024)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automunge](https://mlsanity.com/paper/2202.09248/) (2022)
- [The Paradox of Noise: An Empirical Study of Noise-Infusion Mechanisms to Improve Generalization, Stability, and Privacy in Federated Learning](https://mlsanity.com/paper/2311.05790/) (2023)
- [MeMo: Meaningful, Modular Controllers via Noise Injection](https://mlsanity.com/paper/2407.01567/) (2024)
- [Geometric Regularization from Overparameterization](https://mlsanity.com/paper/2202.09276/) (2022)
- [On gradient descent training under data augmentation with on-line noisy copies](https://mlsanity.com/paper/2206.03734/) (2022)
- [Hyper-Reduced Autoencoders for Efficient and Accurate Nonlinear Model Reductions](https://mlsanity.com/paper/2303.09630/) (2023)
- [Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Ranking](https://mlsanity.com/paper/2502.18478/) (2025)
- [FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddings](https://mlsanity.com/paper/2505.21848/) (2025)
- [Information Hidden in Gradients of Regression with Target Noise](https://mlsanity.com/paper/2601.18546/) (2026)
- [Asymptotic Behavior of Multi--Task Learning: Implicit Regularization and Double Descent Effects](https://mlsanity.com/paper/2603.05060/) (2026)

## Abstract (arXiv)

Randomly perturbing networks during the training process is a commonly used approach to improving generalization performance. In this paper, we present a theoretical study of one particular way of random perturbation, which corresponds to injecting artificial noise to the training data. We provide a precise asymptotic characterization of the training and generalization errors of such randomly perturbed learning problems on a random feature model. Our analysis shows that Gaussian noise injection in the training process is equivalent to introducing a weighted ridge regularization, when the number of noise injections tends to infinity. The explicit form of the regularization is also given. Numerical results corroborate our asymptotic predictions, showing that they are accurate even in moderate problem dimensions. Our theoretical predictions are based on a new correlated Gaussian equivalence conjecture that generalizes recent results in the study of random feature models.