mlsanityData to 7 Oct 2026

Noise Injection as a Probe of Deep Learning Dynamics

arXiv 2210.13599 ↗First posted 24 Oct 2022cs.LG cond-mat.stat-mech cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We propose a new method to probe the learning mechanism of Deep Neural Networks (DNN) by perturbing the system using Noise Injection Nodes (NINs). These nodes inject uncorrelated noise via additional optimizable weights to existing feed-forward network architectures, without changing the optimization algorithm. We find that the system displays distinct phases during training, dictated by the scale of injected noise. We first derive expressions for the dynamics of the network and utilize a simple linear model as a test case. We find that in some cases, the evolution of the noise nodes is similar to that of the unperturbed loss, thus indicating the possibility of using NINs to learn more about the full system in the future.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

24 of 37 references.

References · 24

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  5. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  6. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  7. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  11. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  12. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
12 more
  1. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  2. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  3. 2013Dropout Training as Adaptive Regularizationcited by 89
  4. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  5. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  6. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  7. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  8. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  9. 2016Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networkscited by 19
  10. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18
  11. 2021SGD Implicitly Regularizes Generalization Errorcited by 10
  12. 2016Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networkscited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.

2 papers.

  1. 2025Sparse Neurons Carry Strong Signals of Question Ambiguity in LLMscited by 4
  2. 2022Noise Injection Node Regularization for Robust Learningcited by 1