mlsanityData to 14 Oct 2026

Six Lectures on Linearized Neural Networks

arXiv 2308.13431 ↗First posted 25 Aug 2023stat.ML cs.LG math.STcited by 20

Not on the map. Why?

Abstract (arXiv)

In these six lectures, we examine what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models. We first recall the correspondence between neural networks and linear models via the so-called lazy regime. We then review four models for linearized neural networks: linear regression with concentrated features, kernel ridge regression, random feature model and neural tangent model. Finally, we highlight the limitations of the linear theory and discuss how other approaches can overcome them.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 108 of its 147 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

108 of 147 references.

References · 108

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2017Exploring Generalization in Deep Learningcited by 600
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
38 more
  1. 2019Benign Overfitting in Linear Regressioncited by 549
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  8. 2017Implicit Regularization in Matrix Factorizationcited by 322
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  13. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  14. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  15. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  16. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  17. 2021Deep learning: a statistical viewpointcited by 240
  18. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  19. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  20. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  21. 2018To understand deep learning we need to understand kernel learningcited by 187
  22. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  23. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  24. 2020Directional convergence and alignment in deep learningcited by 178
  25. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  26. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  27. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  28. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  29. 2016Generalization Properties of Learning with Random Featurescited by 164
  30. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  31. 2019Linearized two-layers neural networks in high dimensioncited by 155
  32. 2020Benign overfitting in ridge regressioncited by 153
  33. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  34. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  35. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  36. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  37. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  38. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 3; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  2. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 19
  3. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8
  4. 2024Aligning Model Properties via Conformal Risk Controlcited by 7
  5. 2024Trained quantum neural networks are Gaussian processescited by 6
  6. 2024On the Benefits of Rank in Attention Layerscited by 5
  7. 2025A general technique for approximating high-dimensional empirical kernel matricescited by 4
  8. 2026Pointwise Generalization in Deep Neural Networkscited by 4
  9. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
  10. 2025Provably Learning from Modern Language Models via Low Logit Rankcited by 3
  11. 2025Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scalingcited by 2
  12. 2024Cyclic Group Projection for Enumerating Quasi-Cyclic Codes Trapping Setscited by 1
8 more
  1. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  2. 2026Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networkscited by 1
  3. 2024Topology-Aware Exploration of Energy-Based Models Equilibrium: Toric QC-LDPC Codes and Hyperbolic MET QC-LDPC Codescited by 0
  4. 2024Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regressioncited by 0
  5. 2024Error dynamics of mini-batch gradient descent with random reshuffling for least squares regressioncited by 0
  6. 2024ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networkscited by 0
  7. 2026Sparse Random-Feature Neural Networks with Krylov-Based SVD for Singularly Perturbed ODEcited by 0
  8. 2026Tracing sources of epistemic uncertainty in deep learning predictions: homo- and hetero-scedastic linearized estimatorscited by 0