mlsanityData to 14 Oct 2026

The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks

arXiv 2110.03922 ↗First posted 8 Oct 2021cs.LG stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

We derive simple closed-form estimates for the test risk and other generalization metrics of kernel ridge regression (KRR). Relative to prior work, our derivations are greatly simplified and our final expressions are more readily interpreted. These improvements are enabled by our identification of a sharp conservation law which limits the ability of KRR to learn any orthonormal basis of functions. Test risk and other objects of interest are expressed transparently in terms of our conserved quantity evaluated in the kernel eigenbasis. We use our improved framework to: i) provide a theoretical explanation for the "deep bootstrap" of Nakkiran et al (2020), ii) generalize a previous result regarding the hardness of the classic parity problem, iii) fashion a theoretical tool for the study of adversarial robustness, and iv) draw a tight analogy between KRR and a well-studied system in statistical physics.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 40 of its 84 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

40 of 84 references.

References · 40

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2018On the Spectral Bias of Neural Networkscited by 1,341
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  12. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
28 more
  1. 2021Unsolved Problems in ML Safetycited by 292
  2. 2021Deep learning: a statistical viewpointcited by 240
  3. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  6. 2018Training behavior of deep neural network in frequency domaincited by 174
  7. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  8. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  10. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  11. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  12. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  13. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  14. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  15. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  16. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  17. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  18. 2020Learning Parities with Neural Networkscited by 78
  19. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 69
  20. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  21. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  22. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  23. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 44
  24. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  25. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  26. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  27. 2020Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexitycited by 25
  28. 2021Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 12; 2025: 2; 2026: 2.

26 papers.

Sort:

By area:

  1. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  2. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  3. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  4. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  5. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  6. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  7. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 27
  8. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  9. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 19
  10. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 18
  11. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 15
  12. 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
14 more
  1. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  2. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  3. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  4. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 8
  5. 2024Wilsonian Renormalization of Neural Network Gaussian Processescited by 8
  6. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  7. 2023An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regressioncited by 5
  8. 2024Infinite Width Limits of Self Supervised Neural Networkscited by 4
  9. 2025Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scalingcited by 4
  10. 2024The lazy (NTK) and rich (μP) regimes: a gentle tutorialcited by 2
  11. 2024Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurementcited by 2
  12. 2023Loss Dynamics of Temporal Difference Reinforcement Learningcited by 0
  13. 2026On Kernel Eigen-alignments of KRR: Reconstruction and Generalizationcited by 0
  14. 2026Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theorycited by 0