mlsanityData to 7 Oct 2026

Understanding Why Neural Networks Generalize Well Through GSNR of Parameters

arXiv 2001.07384 ↗First posted 21 Jan 2020cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalize well. In this paper, we provide a novel perspective on these issues using the gradient signal to noise ratio (GSNR) of parameters during training process of DNNs. The GSNR of a parameter is defined as the ratio between its gradient's squared mean and variance, over the data distribution. Based on several approximations, we establish a quantitative relationship between model parameters' GSNR and the generalization gap. This relationship indicates that larger GSNR during training process leads to better generalization performance. Moreover, we show that, different from that of shallow models (e.g. logistic regression, support vector machines), the gradient descent optimization dynamics of DNNs naturally produces large GSNR during training, which is probably the key to DNNs' remarkable generalization ability.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 2; 2024: 10; 2025: 8; 2026: 8.

40 papers.

Sort:

By area:

  1. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124
  2. 2022Dataset Condensation with Contrastive Signalscited by 105
  3. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  4. 2022On the Generalization Mystery in Deep Learningcited by 25
  5. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 24
  6. 2022Deep Learning and Symbolic Regression for Discovering Parametric Equationscited by 22
  7. 2024BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learningcited by 17
  8. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
  9. 2021Robust Optimization for Multilingual Translation with Imbalanced Datacited by 15
  10. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  11. 2025GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigationcited by 11
  12. 2021Disparity Between Batches as a Signal for Early Stoppingcited by 9
28 more
  1. 2023Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameterscited by 9
  2. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  3. 2024A Layer Selection Approach to Test Time Adaptationcited by 7
  4. 2026Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Modelscited by 6
  5. 2024Effect of Ambient-Intrinsic Dimension Gap on Adversarial Vulnerabilitycited by 5
  6. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  7. 2023Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)cited by 3
  8. 2021In Search of Probeable Generalization Measurescited by 2
  9. 2022Exploiting Explainable Metrics for Augmented SGDcited by 2
  10. 2024Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networkscited by 2
  11. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainercited by 2
  12. 2025Per-example gradients: a new frontier for understanding and improving optimizerscited by 2
  13. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  14. 2022Confidence Dimension for Deep Learning based on Hoeffding Inequality and Relative Evaluationcited by 1
  15. 2022Gradient Mask: Lateral Inhibition Mechanism Improves Performance in Artificial Neural Networkscited by 1
  16. 2024CGLearn: Consistent Gradient-Based Learning for Out-of-Distribution Generalizationcited by 1
  17. 2025Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Modelscited by 1
  18. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  19. 2025Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Taskscited by 1
  20. 2026One-Step Generalization Ratio Guided Optimization for Domain Generalizationcited by 1
  21. 2026Drowning in Routine: Signal Dilution in Multi-Turn Agent Trainingcited by 1
  22. 2026Revisiting the Volume Hypothesiscited by 1
  23. 2026Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimensioncited by 1
  24. 2024BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learningcited by 0
  25. 2024Adversarial Vulnerability as a Consequence of On-Manifold Inseparibilitycited by 0
  26. 2025GRADSTOP: Early Stopping of Gradient Descent via Posterior Samplingcited by 0
  27. 2026Evolutionary Search for Automated Design of Uncertainty Quantification Methodscited by 0
  28. 2026Federated Learning for Feature Generalization with Convex Constraintscited by 0