mlsanityData to 7 Oct 2026

Limitations of Lazy Training of Two-layers Neural Networks

arXiv 1906.08899 ↗First posted 21 Jun 2019stat.ML cs.LG math.STcited by 87

Not on the map. Why?

Abstract (arXiv)

We study the supervised learning problem under either of the following two models: (1) Feature vectors boldsymbol xi are d-dimensional Gaussians and responses are yi = f_*(boldsymbol xi) for f_* an unknown quadratic function; (2) Feature vectors boldsymbol xi are distributed as a mixture of two d-dimensional centered Gaussians, and yi's are the corresponding class labels. We use two-layers neural networks with quadratic activations, and compare three different learning regimes: the random features (RF) regime in which we only train the second-layer weights; the neural tangent (NT) regime in which we train a linearization of the neural network around its initialization; the fully trained neural network (NN) regime in which we train all the weights in the network. We prove that, even for the simple quadratic model of point (1), there is a potentially unbounded gap between the prediction risk achieved in these three training regimes, when the number of neurons is smaller than the ambient dimension. When the number of neurons is larger than the number of dimensions, the problem is significantly easier and both NT and NN learning achieve zero risk.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 15; 2023: 25; 2024: 19; 2025: 4; 2026: 3.

87 papers.

Sort:

By area:

  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  3. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  4. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  5. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  6. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  7. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  8. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  9. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  10. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  11. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  12. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
38 more
  1. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  2. 2023Transformers learn through gradual rank increasecited by 52
  3. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  4. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  5. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  6. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  7. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  8. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  9. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  10. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  11. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  12. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  13. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  14. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  15. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  16. 2023Six Lectures on Linearized Neural Networkscited by 19
  17. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  18. 2021A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equationscited by 18
  19. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  20. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  21. 2021Neural Tangent Kernel Maximum Mean Discrepancycited by 15
  22. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  23. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 14
  24. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  25. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  26. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  27. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  28. 2021Relative stability toward diffeomorphisms indicates performance in deep netscited by 12
  29. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
  30. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  31. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  32. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  33. 2023Non-Parametric Representation Learning with Kernelscited by 11
  34. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 10
  35. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 9
  36. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  37. 2021A Recipe for Global Convergence Guarantee in Deep Neural Networkscited by 8
  38. 2021On Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8

50 most cited of 87 All 87 (JSON)