mlsanityData to 7 Oct 2026

Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions

arXiv 2201.08082 ↗First posted 20 Jan 2022stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Empirical observation of high dimensional phenomena, such as the double descent behaviour, has attracted a lot of interest in understanding classical techniques such as kernel methods, and their implications to explain generalization properties of neural networks. Many recent works analyze such models in a certain high-dimensional regime where the covariates are independent and the number of samples and the number of covariates grow at a fixed ratio (i.e. proportional asymptotics). In this work we show that for a large class of kernels, including the neural tangent kernel of fully connected networks, kernel methods can only perform as well as linear models in this regime. More surprisingly, when the data is generated by a kernel model where the relationship between input and the response could be very nonlinear, we show that linear models are in fact optimal, i.e. linear models achieve the minimum risk among all models, linear or nonlinear. These results suggest that more complex models for the data other than independent features are needed for high-dimensional analysis.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

42 of 62 references.

References · 42

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  11. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  12. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
30 more
  1. 2018To understand deep learning we need to understand kernel learningcited by 186
  2. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  3. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  4. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  5. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  6. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  7. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  8. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  9. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  10. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  11. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
  12. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  13. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  14. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  15. 2020The Recurrent Neural Tangent Kernelcited by 67
  16. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  17. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  18. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  19. 2018Entropy and mutual information in models of deep neural networkscited by 50
  20. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  21. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  22. 2020Analytic Study of Double Descent in Binary Classification: The Impact of Losscited by 28
  23. 2020Generalization Error of Generalized Linear Models in High Dimensionscited by 25
  24. 2016Deep Learning and Hierarchal Generative Modelscited by 23
  25. 2020Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)cited by 23
  26. 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
  27. 2019Inference with Deep Generative Priors in High Dimensionscited by 17
  28. 2016Statistical Mechanics of High-Dimensional Inferencecited by 13
  29. 2020Sharp Asymptotics and Optimal Performance for Inference in Binary Modelscited by 13
  30. 2021Asymptotics of Ridge Regression in Convolutional Modelscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 3.

8 papers.

  1. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  2. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 18
  3. 2023Generalized equivalences between subsampling and ridge regularizationcited by 10
  4. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  5. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  6. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  7. 2024Thermodynamic limit in learning period threecited by 1
  8. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1