mlsanityData to 7 Oct 2026

Anticorrelated Noise Injection for Improved Generalization

arXiv 2202.02831 ↗First posted 6 Feb 2022stat.ML cs.LG math.OCcited by 41

Not on the map. Why?

Abstract (arXiv)

Injecting artificial noise into gradient descent (GD) is commonly employed to improve the performance of machine learning models. Usually, uncorrelated noise is used in such perturbed gradient descent (PGD) methods. It is, however, not known if this is optimal or whether other types of noise could provide better generalization performance. In this paper, we zoom in on the problem of correlating the perturbations of consecutive PGD steps. We consider a variety of objective functions for which we find that GD with anticorrelated perturbations ("Anti-PGD") generalizes significantly better than GD and standard (uncorrelated) PGD. To support these experimental findings, we also derive a theoretical analysis that demonstrates that Anti-PGD moves to wider minima, while GD and PGD remain stuck in suboptimal regions or even diverge. This new connection between anticorrelated noise and generalization opens the field to novel ways to exploit noise for training machine learning models.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 6 Feb 2022.

Builds on

32 of 45 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2017Exploring Generalization in Deep Learningcited by 595
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  9. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  10. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  11. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
20 more
  1. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  2. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  3. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  4. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  5. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  6. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  7. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  8. 2020Relative Flatness and Generalizationcited by 88
  9. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  10. 2017Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priorscited by 81
  11. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  12. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  13. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  14. 2017Catalyst Acceleration for First-order Convex Optimization: from Theory to Practicecited by 39
  15. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  16. 2019Non-Gaussianity of Stochastic Gradient Noisecited by 22
  17. 2019Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processescited by 20
  18. 2019How noise affects the Hessian spectrum in overparameterized neural networkscited by 17
  19. 2018Identifying Generalization Properties in Neural Networkscited by 15
  20. 2021Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noisecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 19; 2024: 5; 2025: 6; 2026: 5.

41 papers.

Sort:

  1. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  2. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  3. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  4. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  5. 2024Why is SAM Robust to Label Noise?cited by 21
  6. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  7. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  8. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  9. 2023A Study on FGSM Adversarial Training for Neural Retrievalcited by 10
  10. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  11. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  12. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averagingcited by 9
29 more
  1. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  2. 2023Data Optimization in Deep Learning: A Surveycited by 6
  3. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
  4. 2023Utility-based Perturbed Gradient Descent: An Optimizer for Continual Learningcited by 5
  5. 2023Edge Intelligence Over the Air: Two Faces of Interference in Federated Learningcited by 5
  6. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  7. 2025SAFE: Finding Sparse and Flat Minima to Improve Pruningcited by 5
  8. 2023ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compressioncited by 4
  9. 2025Staying on the Manifold: Geometry-Aware Noise Injectioncited by 4
  10. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  11. 2023The Lie-Group Bayesian Learning Rulecited by 3
  12. 2023On the curvature of the loss landscapecited by 3
  13. 2023Enhancing Deep Neural Network Training Efficiency and Performance through Linear Predictioncited by 3
  14. 2023Critical Influence of Overparameterization on Sharpness-aware Minimizationcited by 3
  15. 2022Disentangling the Mechanisms Behind Implicit Regularization in SGDcited by 2
  16. 2023The Split Matters: Flat Minima Methods for Improving the Performance of GNNscited by 2
  17. 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
  18. 2024Communication-Efficient Distributed Learning with Local Immediate Error Compensationcited by 2
  19. 2025Generalization Bounds for Markov Algorithms through Entropy Flow Computationscited by 2
  20. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  21. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  22. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  23. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  24. 2023Evolutionary Algorithms in the Light of SGD: Limit Equivalence, Minima Flatness, and Transfer Learningcited by 1
  25. 2024SGD Jittering: A Training Strategy for Robust and Accurate Model-Based Architecturescited by 0
  26. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0
  27. 2026Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descentcited by 0
  28. 2026Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Modelscited by 0
  29. 2026Fast PAC Global Optimization via Restarted Langevin: Exploration, Exploitation, and Degenerate Coolingcited by 0