mlsanityData to 7 Oct 2026

Improving Adaptivity via Over-Parameterization in Sequence Models

arXiv 2409.00894 ↗First posted 2 Sep 2024cs.LG stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

It is well known that eigenfunctions of a kernel play a crucial role in kernel regression. Through several examples, we demonstrate that even with the same set of eigenfunctions, the order of these functions significantly impacts regression outcomes. Simplifying the model by diagonalizing the kernel, we introduce an over-parameterized gradient descent in the realm of sequence model to capture the effects of various orders of a fixed set of eigen-functions. This method is designed to explore the impact of varying eigenfunction orders. Our theoretical results show that the over-parameterization gradient flow can adapt to the underlying structure of the signal and significantly outperform the vanilla gradient flow method. Moreover, we also demonstrate that deeper over-parameterization can further enhance the generalization capability of the model. These results not only provide a new perspective on the benefits of over-parameterization and but also offer insights into the adaptivity and generalization potential of neural networks beyond the kernel regime.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

37 of 52 references.

References · 37

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  11. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
25 more
  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  3. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  4. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  5. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  6. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  7. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  8. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  9. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  10. 2021Implicit Regularization in Tensor Factorizationcited by 43
  11. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  12. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  13. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  14. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  15. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  16. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  17. 2023On the Optimality of Misspecified Spectral Algorithmscited by 28
  18. 2019High-Dimensional Linear Regression via Implicit Regularizationcited by 25
  19. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  20. 2020Understanding Implicit Regularization in Over-Parameterized Single Index Modelcited by 12
  21. 2024On the Saturation Effect of Kernel Ridge Regressioncited by 12
  22. 2019Implicit Regularization in Over-parameterized Neural Networkscited by 10
  23. 2024Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decaycited by 8
  24. 2023A Theory of Feature Learning in Kernel Modelscited by 3
  25. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
  2. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0