mlsanityData to 7 Oct 2026

Generalization in Supervised Learning Through Riemannian Contraction

arXiv 2201.06656 ↗First posted 17 Jan 2022cs.LG math.DScited by 8

Not on the map. Why?

Abstract (arXiv)

We prove that Riemannian contraction in a supervised learning setting implies generalization. Specifically, we show that if an optimizer is contracting in some Riemannian metric with rate λ > 0, it is uniformly algorithmically stable with rate O(1/λ n), where n is the number of labelled examples in the training set. The results hold for stochastic and deterministic optimization, in both continuous and discrete-time, for convex and non-convex loss surfaces. The associated generalization bounds reduce to well-known results in the particular case of gradient descent over convex or strongly convex loss surfaces. They can be shown to be optimal in certain linear settings, such as kernel ridge regression under gradient flow.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 54 references that are in the map. First posted on arXiv 17 Jan 2022.

Builds on

28 of 54 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  8. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  9. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  10. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  11. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  12. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
16 more
  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  3. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
  4. 2019Sharper bounds for uniformly stable algorithmscited by 91
  5. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  6. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  7. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  8. 2020Regret Bounds for Adaptive Nonlinear Controlcited by 25
  9. 2018Beyond Convexity -- Contraction and Global Convergence of Gradient Descentcited by 17
  10. 2021Algorithmic Instabilities of Accelerated Gradient Descentcited by 16
  11. 2020Instability, Computational Efficiency and Statistical Accuracycited by 12
  12. 2018On the choice of metric in gradient-based theories of brain functioncited by 8
  13. 2018A continuous-time analysis of distributed stochastic gradientcited by 4
  14. 2021Adversarially Robust Stability Certificates can be Sample-Efficientcited by 2
  15. 2013Stochastic Contraction in Riemannian Metricscited by 1
  16. 2013Contraction analysis of nonlinear random dynamical systemscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 4; 2024: 0; 2025: 1.

8 papers.

  1. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  2. 2022Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GDcited by 19
  3. 2022Generalization Bounds for Stochastic Gradient Descent via Localized varepsilon-Coverscited by 11
  4. 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
  5. 2023Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descentcited by 6
  6. 2023Generalization Bounds for Label Noise Stochastic Gradient Descentcited by 3
  7. 2023Behavior quantification as the missing link between fields: Tools for digital psychiatry and their role in the future of neurobiologycited by 1
  8. 2025Descend or Rewind? Stochastic Gradient Descent Unlearningcited by 0