mlsanityData to 7 Oct 2026

Explicit Regularization in Overparametrized Models via Noise Injection

arXiv 2206.04613 ↗First posted 9 Jun 2022cs.LG stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

Injecting noise within gradient descent has several desirable features, such as smoothing and regularizing properties. In this paper, we investigate the effects of injecting noise before computing a gradient step. We demonstrate that small perturbations can induce explicit regularization for simple models based on the L1-norm, group L1-norms, or nuclear norms. However, when applied to overparametrized neural networks with large widths, we show that the same perturbations can cause variance explosion. To overcome this, we propose using independent layer-wise perturbations, which provably allow for explicit regularization without variance explosion. Our empirical results show that these small perturbations lead to improved generalization performance compared to vanilla gradient descent.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 42 of its 69 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

42 of 69 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  11. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  12. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
30 more
  1. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  2. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  3. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  4. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  5. 2017Implicit Regularization in Matrix Factorizationcited by 318
  6. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2017How to Escape Saddle Points Efficientlycited by 284
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  11. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  12. 2016Tensor Decomposition for Signal Processing and Machine Learningcited by 223
  13. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  14. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  15. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  16. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  17. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  18. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  19. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  20. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  21. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  22. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  23. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  24. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  25. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  26. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  27. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  28. 2017A Generic Approach for Escaping Saddle pointscited by 13
  29. 2019Implicit Regularization in Over-parameterized Neural Networkscited by 10
  30. 2020Explicit Regularization of Stochastic Gradient Methods through Dualitycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 8; 2025: 9; 2026: 1.

25 papers.

Sort:

By area:

  1. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  2. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  3. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  4. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
  5. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  6. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  7. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  8. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  9. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  10. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
  11. 2025Staying on the Manifold: Geometry-Aware Noise Injectioncited by 4
  12. 2024Regularized Gauss-Newton for Optimizing Overparameterized Neural Networkscited by 3
13 more
  1. 2024Demystifying SGD with Doubly Stochastic Gradientscited by 2
  2. 2025Generalization Bounds for Markov Algorithms through Entropy Flow Computationscited by 2
  3. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  4. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  5. 2023Why is parameter averaging beneficial in SGD? An objective smoothing perspectivecited by 0
  6. 2024Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithmscited by 0
  7. 2024Performance Evaluation of Knowledge Graph Embedding Approaches under Non-adversarial Attackscited by 0
  8. 2024SGD Jittering: A Training Strategy for Robust and Accurate Model-Based Architecturescited by 0
  9. 2025Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Rankingcited by 0
  10. 2025Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injectioncited by 0
  11. 2025High-order Regularization for Machine Learning and Learning-based Controlcited by 0
  12. 2025Balancing Utility and Privacy: Dynamically Private SGD with Random Projectioncited by 0
  13. 2026Information Hidden in Gradients of Regression with Target Noisecited by 0