mlsanityData to 7 Oct 2026

Handbook of Convergence Theorems for (Stochastic) Gradient Methods

arXiv 2301.11235 ↗First posted 26 Jan 2023math.OCcited by 158

Not on the map. Why?

Abstract (arXiv)

This is a handbook of simple proofs of the convergence of gradient and stochastic gradient descent type methods. We consider functions that are Lipschitz, smooth, convex, strongly convex, and/or Polyak-Lojasiewicz functions. Our focus is on ``good proofs'' that are also simple. Each section can be consulted separately. We start with proofs of gradient descent, then on stochastic variants, including minibatching and momentum. Then move on to nonsmooth problems with the subgradient method, the proximal gradient descent and their stochastic variants. Our focus is on global convergence rates and complexity rates. Some slightly less common proofs found here include that of SGD (Stochastic gradient descent) with a proximal step, with momentum, and with mini-batching without replacement.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jan 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 44; 2025: 57; 2026: 32.

158 papers.

Sort:

By area:

  1. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  2. 2023Provable convergence guarantees for black-box variational inferencecited by 29
  3. 2023MoMo: Momentum Models for Adaptive Learning Ratescited by 24
  4. 2023SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identitycited by 23
  5. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  6. 2024Provable Benefit of Annealed Langevin Monte Carlo for Non-log-concave Samplingcited by 23
  7. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  8. 2023Aiming towards the minimizers: fast convergence of SGD for overparametrized problemscited by 22
  9. 2024Batch and match: black-box variational inference with a score-based divergencecited by 22
  10. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  11. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  12. 2023On the Sample Complexity of Quantum Boltzmann Machine Learningcited by 18
38 more
  1. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  2. 2024Plastic Learning with Deep Fourier Featurescited by 14
  3. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 13
  4. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  5. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  6. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 13
  7. 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
  8. 2024On the Complexity of First-Order Methods in Stochastic Bilevel Optimizationcited by 11
  9. 2024Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussianscited by 11
  10. 2024Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descentcited by 11
  11. 2024Gradient Methods with Online Scalingcited by 11
  12. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  13. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  14. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  15. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  16. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 9
  17. 2024Rethinking Weight Decay for Robust Fine-Tuning of Foundation Modelscited by 9
  18. 2025When to Forget? Complexity Trade-offs in Machine Unlearningcited by 9
  19. 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Modelscited by 9
  20. 2025On the Convergence Rate of LoRA Gradient Descentcited by 9
  21. 2023Curvature-Aware Training for Coordinate Networkscited by 8
  22. 2023Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERMcited by 8
  23. 2024Spatiotemporal Covariance Neural Networkscited by 8
  24. 2024Adaptive Non-uniform Timestep Sampling for Accelerating Diffusion Model Trainingcited by 8
  25. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 7
  26. 2025Sharp higher order convergence rates for the Adam optimizercited by 7
  27. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7
  28. 2025Deep Unfolding: Recent Developments, Theory, and Design Guidelinescited by 7
  29. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 7
  30. 2023Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descentcited by 6
  31. 2024Fully Zeroth-Order Bilevel Programming via Gaussian Smoothingcited by 6
  32. 2024MGDA Converges under Generalized Smoothness, Provablycited by 6
  33. 2024Efficient Statistics With Unknown Truncation, Polynomial Time Algorithms, Beyond Gaussianscited by 6
  34. 2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
  35. 2025When, Where and Why to Average Weights?cited by 6
  36. 2024Mitigating Privacy Risk in Membership Inference by Convex-Concave Losscited by 5
  37. 2024Almost sure convergence rates of stochastic gradient methods under gradient dominationcited by 5
  38. 2024BadGD: A unified data-centric framework to identify gradient descent vulnerabilitiescited by 5

50 most cited of 158 All 158 (JSON)