mlsanityData to 7 Oct 2026

On the Inherent Regularization Effects of Noise Injection During Training

arXiv 2102.07379 ↗First posted 15 Feb 2021cs.LG cs.IT math.ITcited by 18

Not on the map. Why?

Abstract (arXiv)

Randomly perturbing networks during the training process is a commonly used approach to improving generalization performance. In this paper, we present a theoretical study of one particular way of random perturbation, which corresponds to injecting artificial noise to the training data. We provide a precise asymptotic characterization of the training and generalization errors of such randomly perturbed learning problems on a random feature model. Our analysis shows that Gaussian noise injection in the training process is equivalent to introducing a weighted ridge regularization, when the number of noise injections tends to infinity. The explicit form of the regularization is also given. Numerical results corroborate our asymptotic predictions, showing that they are accurate even in moderate problem dimensions. Our theoretical predictions are based on a new correlated Gaussian equivalence conjecture that generalizes recent results in the study of random feature models.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

21 of 43 references.

References · 21

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  3. 2018Adversarial Logit Pairingcited by 224
  4. 2019Generalized Sliced Wasserstein Distancescited by 205
  5. 2018To understand deep learning we need to understand kernel learningcited by 186
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  7. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  8. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  9. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  10. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  11. 2016Noisy Activation Functionscited by 53
  12. 2017Efficient Defenses Against Adversarial Attackscited by 53
9 more
  1. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  2. 2013A framework to characterize performance of LASSO algorithmscited by 38
  3. 2019The Impact of Regularization on High-dimensional Logistic Regressioncited by 35
  4. 2020MaxUp: A Simple Way to Improve Generalization of Neural Network Trainingcited by 27
  5. 2014Analyzing noise in autoencoders and deep networkscited by 25
  6. 2019The Implicit Regularization of Ordinary Least Squares Ensemblescited by 25
  7. 2015Universality laws for randomized dimension reduction, with applicationscited by 21
  8. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 21
  9. 2017Symbol Error Rate Performance of Box-relaxation Decoders in Massive MIMOcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 4; 2024: 4; 2025: 3; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  2. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  3. 2022On how to avoid exacerbating spurious correlations when models are overparameterizedcited by 4
  4. 2023Least Squares Regression Can Exhibit Under-Parameterized Double Descentcited by 4
  5. 2024A Novel Gaussian Min-Max Theorem and its Applicationscited by 4
  6. 2022Noise Injection as a Probe of Deep Learning Dynamicscited by 2
  7. 2024Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalancecited by 2
  8. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  9. 2022Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automungecited by 1
  10. 2023The Paradox of Noise: An Empirical Study of Noise-Infusion Mechanisms to Improve Generalization, Stability, and Privacy in Federated Learningcited by 1
  11. 2024MeMo: Meaningful, Modular Controllers via Noise Injectioncited by 1
  12. 2022Geometric Regularization from Overparameterizationcited by 0
6 more
  1. 2022On gradient descent training under data augmentation with on-line noisy copiescited by 0
  2. 2023Hyper-Reduced Autoencoders for Efficient and Accurate Nonlinear Model Reductionscited by 0
  3. 2025Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Rankingcited by 0
  4. 2025FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddingscited by 0
  5. 2026Information Hidden in Gradients of Regression with Target Noisecited by 0
  6. 2026Asymptotic Behavior of Multi--Task Learning: Implicit Regularization and Double Descent Effectscited by 0