Data to 7 Oct 2026

Train faster, generalize better: Stability of stochastic gradient descent

arXiv 1509.01240 ↗First posted 3 Sep 2015cs.LG math.OC stat.MLcited by 547

Not on the map. Why?

Abstract (arXiv)

We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions. Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.

Key facts. Cited by 547 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Sep 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 97; 2022: 109; 2023: 90; 2024: 71; 2025: 85; 2026: 94.

547 papers.

Sort:

By area:

2023Detecting Pretraining Data from Large Language Modelscited by 375
2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
2021Training Neural Networks with Fixed Sparse Maskscited by 212
2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
2021Private Stochastic Convex Optimization: Optimal Rates in ℓ1 Geometrycited by 83
2021Topology-Aware Segmentation Using Discrete Morse Theorycited by 75
2021On the geometry of generalization and memorization in deep neural networkscited by 71
2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61

38 more

2022Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Losscited by 60
2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
2024Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearningcited by 48
2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
2022Private Convex Optimization via Exponential Mechanismcited by 47
2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
2021Stability and Deviation Optimal Risk Bounds with Convergence Rate O(1/n)cited by 42
2021Subspace Adversarial Trainingcited by 41
2023Towards Understanding the Generalization of Graph Neural Networkscited by 41
2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 39
2021SGD Generalizes Better Than GD (And Regularization Doesn't Help)cited by 38
2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 37
2022Topology-aware Generalization of Decentralized SGDcited by 37
2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
2021Leave-one-out Unfairnesscited by 36
2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
2021Stability and Generalization of Stochastic Gradient Methods for Minimax Problemscited by 35
2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 35
2021Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraintscited by 34
2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
2021Deep Active Learning by Leveraging Training Dynamicscited by 32
2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
2021Private Federated Learning Without a Trusted Server: Optimal Algorithms for Convex Lossescited by 30
2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 29
2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
2022Stability vs Implicit Bias of Gradient Methods on Separable Data and Beyondcited by 29
2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27

50 most cited of 547 All 547 (JSON)