mlsanityData to 7 Oct 2026

Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods

arXiv 2205.14818 ↗First posted 30 May 2022stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

While deep learning has outperformed other methods for various tasks, theoretical frameworks that explain its reason have not been fully established. To address this issue, we investigate the excess risk of two-layer ReLU neural networks in a teacher-student regression model, in which a student network learns an unknown teacher network through its outputs. Especially, we consider the student network that has the same width as the teacher network and is trained in two phases: first by noisy gradient descent and then by the vanilla gradient descent. Our result shows that the student network provably reaches a near-global optimal solution and outperforms any kernel methods estimator (more generally, linear estimators), including neural tangent kernel approach, random feature model, and other kernel methods, in a sense of the minimax optimal rate. The key concept inducing this superiority is the non-convexity of the neural network models. Even though the loss landscape is highly non-convex, the student network adaptively learns the teacher neurons.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 48 of its 59 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

48 of 59 references.

References · 48

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  11. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  12. 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 224
36 more
  1. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  2. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  3. 2019Linearized two-layers neural networks in high dimensioncited by 155
  4. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  9. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  10. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  11. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  12. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  13. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  14. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  15. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  16. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  17. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  18. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  19. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  20. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  21. 2020Learning a Single Neuron with Gradient Methodscited by 50
  22. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  23. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 49
  24. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  25. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  26. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  27. 2018Global Non-convex Optimization with Discretized Diffusionscited by 44
  28. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  29. 2019On the minimax optimality and superiority of deep neural network learning over sparse parameter spacescited by 24
  30. 2020Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysiscited by 21
  31. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  32. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  33. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  34. 2021Analysis of feature learning in weight-tied autoencoders via the mean field lenscited by 13
  35. 2020Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methodscited by 9
  36. 2020Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamicscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 1.

10 papers.

  1. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 15
  2. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  3. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  4. 2023Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descentcited by 6
  5. 2024Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clippingcited by 3
  6. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  7. 2025Sobolev acceleration for neural networkscited by 3
  8. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  9. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Pointcited by 1
  10. 2026Variance Reduction for Stochastic Gradient Generalized Non-reversible Langevin Monte Carlo Algorithmscited by 1