mlsanityData to 14 Oct 2026

Convergence beyond the over-parameterized regime using Rayleigh quotients

arXiv 2301.08117 ↗First posted 19 Jan 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a new strategy to prove the convergence of deep learning architectures to a zero training (or even testing) loss by gradient flow. Our analysis is centered on the notion of Rayleigh quotients in order to prove Kurdyka-Lojasiewicz inequalities for a broader set of neural network architectures and loss functions. We show that Rayleigh quotients provide a unified view for several convergence analysis techniques in the literature. Our strategy produces a proof of convergence for various examples of parametric learning. In particular, our analysis does not require the number of parameters to tend to infinity, nor the number of samples to be finite, thus extending to test loss minimization and beyond the over-parameterized regime.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 15 of its 38 references that are in the map. First posted on arXiv 19 Jan 2023.

Builds on

15 of 38 references.

References · 15

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2021Convex optimizationcited by 2,515
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  9. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  10. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  11. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  12. 2018Convergence of Gradient Descent on Separable Datacited by 110
3 more
  1. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 88
  2. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  3. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2023Rethinking Gauss-Newton for learning over-parameterized modelscited by 6
  2. 2024Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems trained with Gradient Descentcited by 1
  3. 2025Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networkscited by 0
  4. 2026Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Casecited by 0
  5. 2026Sustaining Plasticity via Learnable Wavelet Activations in Continual Learningcited by 0