mlsanityData to 14 Oct 2026

On the interplay between noise and curvature and its effect on optimization and generalization

arXiv 1906.07774 ↗First posted 18 Jun 2019cs.LG stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the variance of the gradients. While most previous works focus on one or the other of these properties, we explore how their interaction affects optimization speed. Further, as the ultimate goal is good generalization performance, we clarify how both curvature and noise are relevant to properly estimate the generalization gap. Realizing that the limitations of some existing works stems from a confusion between these matrices, we also clarify the distinction between the Fisher matrix, the Hessian, and the covariance matrix of the gradients.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 10; 2023: 10; 2024: 2; 2025: 5; 2026: 6.

44 papers.

Sort:

By area:

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  3. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  4. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  5. 2021Strength of Minibatch Noise in SGDcited by 39
  6. 2021Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscited by 19
  7. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  8. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  9. 2025Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisitioncited by 19
  10. 2022Hybrid quantum ResNet for car classification and its hyperparameter optimizationcited by 17
  11. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  12. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 16
32 more
  1. 2021Robust Optimization for Multilingual Translation with Imbalanced Datacited by 15
  2. 2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
  3. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 14
  4. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  5. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  6. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 9
  7. 2022A Mini-Block Fisher Method for Deep Neural Networkscited by 8
  8. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  9. 2022Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioningcited by 7
  10. 2021Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Taskscited by 5
  11. 2023Optimizer's Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimizationcited by 4
  12. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  13. 2021L2M: Practical posterior Laplace approximation with optimization-driven second moment estimationcited by 2
  14. 2021A generalization gap estimation for overparameterized models via the Langevin functional variancecited by 2
  15. 2022Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passingcited by 2
  16. 2022Noise Injection as a Probe of Deep Learning Dynamicscited by 2
  17. 2023Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directionscited by 2
  18. 2023Fading memory as inductive bias in residual recurrent networkscited by 2
  19. 2023Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorizationcited by 2
  20. 2023SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscapecited by 1
  21. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  22. 2024CurvFed: Curvature-Aligned Federated Learning for Fairness without Demographicscited by 1
  23. 2025Flatness After All?cited by 1
  24. 2026Gradient Regularized Natural Gradientscited by 1
  25. 2026Ethical Fairness in Ubiquitous Health Sensing without Known Attributescited by 1
  26. 2026Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuationscited by 1
  27. 2022How to select an objective function using information theorycited by 0
  28. 2025A Bootstrap Perspective on Stochastic Gradient Descentcited by 0
  29. 2025TPV: Parameter Perturbations Through the Lens of Test Prediction Variancecited by 0
  30. 2026Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regimecited by 0
  31. 2026On-Average Stability of Multipass Preconditioned SGD and Effective Dimensioncited by 0
  32. 2026Rethinking Bregman Divergences in Kronecker-Factored Optimizerscited by 0