mlsanityData to 14 Oct 2026

How rotational invariance of common kernels prevents generalization in high dimensions

arXiv 2104.04244 ↗First posted 9 Apr 2021math.ST cs.LG stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

Kernel ridge regression is well-known to achieve minimax optimal rates in low-dimensional settings. However, its behavior in high dimensions is much less understood. Recent work establishes consistency for kernel regression under certain assumptions on the ground truth function and the distribution of the input data. In this paper, we show that the rotational invariance property of commonly studied kernels (such as RBF, inner product kernels and fully-connected NTK of any depth) induces a bias towards low-degree polynomials in high dimensions. Our result implies a lower bound on the generalization error for a wide range of distributions and various choices of the scaling for kernels with different eigenvalue decays. This lower bound suggests that general consistency results for kernel ridge regression in high dimensions require a more refined analysis that depends on the structure of the kernel beyond its eigenvalue decay.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 22 of its 48 references that are in the map. First posted on arXiv 9 Apr 2021.

Builds on

22 of 48 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Deep Neural Networks as Gaussian Processescited by 622
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  8. 2018To understand deep learning we need to understand kernel learningcited by 187
  9. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  12. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
10 more
  1. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  2. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  3. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  4. 2018Does data interpolation contradict statistical optimality?cited by 95
  5. 2021Learning with invariances in random features and kernel modelscited by 89
  6. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  7. 2020Neural Kernels Without Tangentscited by 56
  8. 2015Explaining the Success of AdaBoost and Random Forests as Interpolating Classifierscited by 45
  9. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  10. 2017Kernel Feature Selection via Conditional Covariance Minimizationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 5; 2024: 7; 2025: 10; 2026: 2.

32 papers.

Sort:

By area:

  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  3. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  4. 2021What can linearized neural networks actually say about generalization?cited by 49
  5. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  6. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  7. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  8. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  9. 2022The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspectivecited by 24
  10. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  11. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  12. 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
20 more
  1. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 11
  2. 2024Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensionscited by 11
  3. 2023Strong inductive biases provably prevent harmless interpolationcited by 10
  4. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 10
  5. 2024The phase diagram of kernel interpolation in large dimensionscited by 9
  6. 2022Overparameterized random feature regression with nearly orthogonal datacited by 7
  7. 2021On the Double Descent of Random Features Models Trained with SGDcited by 5
  8. 2025Learning Curves of Stochastic Gradient Descent in Kernel Regressioncited by 4
  9. 2025A general technique for approximating high-dimensional empirical kernel matricescited by 4
  10. 2026Power-Law Spectrum of the Random Feature Modelcited by 4
  11. 2025Test time training enhances in-context learning of nonlinear functionscited by 3
  12. 2025Transfer Learning for Benign Overfitting in High-Dimensional Linear Regressioncited by 3
  13. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 2
  14. 2025On the Saturation Effects of Spectral Algorithms in Large Dimensionscited by 2
  15. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1
  16. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  17. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  18. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  19. 2025IIKL: Isometric Immersion Kernel Learning with Riemannian Manifold for Geometric Preservationcited by 0
  20. 2026Perturbative methods for non-parametric instrumental variablecited by 0