mlsanityData to 14 Oct 2026

On Linear Stability of SGD and Input-Smoothness of Neural Networks

arXiv 2105.13462 ↗First posted 27 May 2021cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data. By this connection, it is shown that flat minima regularize the gradient of the model function, which explains the good generalization performance of flat minima. Then, we go beyond the flatness and consider high-order moments of the gradient noise, and show that Stochastic Gradient Descent (SGD) tends to impose constraints on these moments by a linear stability analysis of SGD around global minima. Together with the multiplicative structure, we identify the Sobolev regularization effect of SGD, i.e. SGD regularizes the Sobolev seminorms of the model function with respect to the input data. Finally, bounds for generalization error and adversarial robustness are provided for solutions found by SGD under assumptions of the data distribution.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 27 of its 54 references that are in the map. First posted on arXiv 27 May 2021.

Builds on

27 of 54 references.

References · 27

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  6. 2017Exploring Generalization in Deep Learningcited by 600
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  8. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  9. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  10. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  11. 2017Three Factors Influencing Minima in SGDcited by 273
  12. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
15 more
  1. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  2. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  3. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  4. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 133
  5. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  6. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  7. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  8. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  9. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  10. 2018On exponential convergence of SGD in non-convex over-parametrized learningcited by 33
  11. 2017Gradient Regularization Improves Accuracy of Discriminative Modelscited by 26
  12. 2020On Uniform Convergence and Low-Norm Interpolation Learningcited by 24
  13. 2018Lipschitz regularized Deep Neural Networks generalize and are adversarially robustcited by 19
  14. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16
  15. 2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 18; 2024: 10; 2025: 12; 2026: 15.

63 papers.

Sort:

By area:

  1. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  2. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  4. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  5. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  6. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  7. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  8. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  9. 2022A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspectivecited by 27
  10. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  11. 2025Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptivecited by 20
  12. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 18
38 more
  1. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  2. 2023Loss Spike in Training Neural Networkscited by 16
  3. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  4. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  5. 2026A Defense of the Quadratic Modelcited by 14
  6. 2023How to escape sharp minima with random perturbationscited by 13
  7. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  8. 2023Understanding Grokking Through A Robustness Viewpointcited by 12
  9. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  10. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  11. 2023Exact Mean Square Linear Stability Analysis for SGDcited by 11
  12. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  13. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  14. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  15. 2022Deep Double Descent via Smooth Interpolationcited by 8
  16. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
  17. 2023On progressive sharpening, flat minima and generalisationcited by 7
  18. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  19. 2024Where Do Large Learning Rates Lead Us?cited by 6
  20. 2026The Origin of Edge of Stabilitycited by 6
  21. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  22. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  23. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  24. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  25. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  26. 2024Simplicity Bias via Global Convergence of Sharpness Minimizationcited by 4
  27. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 4
  28. 2025A Dynamical Systems Perspective on the Analysis of Neural Networkscited by 3
  29. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 3
  30. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 3
  31. 2023Selective Volume Mixup for Video Action Recognitioncited by 2
  32. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2
  33. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  34. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  35. 2026Generalization at the Edge of Stabilitycited by 2
  36. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  37. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  38. 2023A simple connection from loss flatness to compressed neural representationscited by 0

50 most cited of 63 All 63 (JSON)