mlsanityData to 7 Oct 2026

Kernel regression in high dimensions: Refined analysis beyond double descent

arXiv 2010.02681 ↗First posted 6 Oct 2020stat.ML cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide a precise characterization of generalization properties of high dimensional kernel ridge regression across the under- and over-parameterized regimes, depending on whether the number of training data n exceeds the feature dimension d. By establishing a bias-variance decomposition of the expected excess risk, we show that, while the bias is (almost) independent of d and monotonically decreases with n, the variance depends on n, d and can be unimodal or monotonically decreasing under different regularization schemes. Our refined analysis goes beyond the double descent theory by showing that, depending on the data eigen-profile and the level of regularization, the kernel regression risk curve can be a double-descent-like, bell-shaped, or monotonic function of n. Experiments on synthetic and real data are conducted to support our theoretical findings.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 9; 2023: 9; 2024: 7; 2025: 1; 2026: 1.

39 papers.

Sort:

  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  2. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  3. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  4. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  5. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  6. 2022On the Impossible Safety of Large AI Modelscited by 27
  7. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  8. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  9. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  10. 2023Six Lectures on Linearized Neural Networkscited by 19
  11. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  12. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
27 more
  1. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  2. 2021Understanding Generalization in Adversarial Training via the Bias-Variance Decompositioncited by 10
  3. 2022Multiple Descent in the Multiple Random Feature Modelcited by 10
  4. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  5. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  6. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  7. 2023Strong inductive biases provably prevent harmless interpolationcited by 9
  8. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 9
  9. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  10. 2023A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regressioncited by 8
  11. 2024Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decaycited by 8
  12. 2021Random matrices in service of ML footprint: ternary random features with no performance losscited by 7
  13. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  14. 2021Double-descent curves in neural networks: a new perspective using Gaussian processescited by 5
  15. 2021On the Double Descent of Random Features Models Trained with SGDcited by 5
  16. 2021Adversarially Robust Kernel Smoothingcited by 4
  17. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  18. 2024High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularizationcited by 4
  19. 2024"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approachcited by 3
  20. 2022Benefit of Interpolation in Nearest Neighbor Algorithmscited by 1
  21. 2023Adversarial Training with Generated Data in High-Dimensional Regression: An Asymptotic Studycited by 1
  22. 2023Changing the Kernel During Training Leads to Double Descent in Kernel Regressioncited by 1
  23. 2025High-dimensional Nonparametric Contextual Bandit Problemcited by 1
  24. 2022A Generalized Weighted Optimization Method for Computational Learning and Inversioncited by 0
  25. 2024Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the role of model complexitycited by 0
  26. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  27. 2026A Rigorous, Tractable Measure of Model Complexitycited by 0