mlsanityData to 7 Oct 2026

Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections

arXiv 2102.07006 ↗First posted 13 Feb 2021stat.ML cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian noise to the network activations at every iteration of the optimisation algorithm, which is typically chosen as stochastic gradient descent (SGD). In this paper we focus on the so-called `implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of SGD. We show that this effect induces an asymmetric heavy-tailed noise on SGD gradient updates. In order to model this modified dynamics, we first develop a Langevin-like stochastic differential equation that is driven by a general family of asymmetric heavy-tailed noise. Using this model we then formally prove that GNIs induce an `implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry. Our empirical results confirm that different types of neural networks trained with GNIs are well-modelled by the proposed dynamics and that the implicit effect of these injections induces a bias that degrades the performance of networks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 24 of its 69 references that are in the map. First posted on arXiv 13 Feb 2021.

Builds on

24 of 69 references.

References · 24

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  3. 2016An overview of gradient descent optimization algorithmscited by 1,021
  4. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  8. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 195
  9. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  10. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  11. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  12. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
12 more
  1. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  2. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  3. 2019On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networkscited by 42
  4. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  5. 2020Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noisecited by 33
  6. 2018Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Accelerationcited by 31
  7. 2014Analyzing noise in autoencoders and deep networkscited by 25
  8. 2017Fractional Langevin Monte Carlo: Exploring L'evy Driven Stochastic Differential Equations for Markov Chain Monte Carlocited by 12
  9. 2018Breaking Reversibility Accelerates Langevin Dynamics for Global Non-Convex Optimizationcited by 12
  10. 2018Noisin: Unbiased Regularization for Recurrent Neural Networkscited by 8
  11. 2020Fractional moment-preserving initialization schemes for training deep neural networkscited by 3
  12. 2015Features modeling with an α-stable distribution: Application to pattern recognition based on continuous belief functionscited by 2

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 2; 2023: 4; 2024: 1; 2025: 1.

12 papers.

  1. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  2. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  3. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  4. 2022Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variancecited by 22
  5. 2023Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptationcited by 20
  6. 2022Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descentcited by 9
  7. 2023Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspectivecited by 5
  8. 2023Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGDcited by 5
  9. 2023High Probability Analysis for Non-Convex Stochastic Optimization with Clippingcited by 5
  10. 2024Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tailscited by 4
  11. 2021Variational Autoencoders: A Harmonic Perspectivecited by 0
  12. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0