mlsanityData to 7 Oct 2026

On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting

arXiv 2106.06251 ↗First posted 11 Jun 2021stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Deep learning empirically achieves high performance in many applications, but its training dynamics has not been fully understood theoretically. In this paper, we explore theoretical analysis on training two-layer ReLU neural networks in a teacher-student regression model, in which a student network learns an unknown teacher network through its outputs. We show that with a specific regularization and sufficient over-parameterization, the student network can identify the parameters of the teacher network with high probability via gradient descent with a norm dependent stepsize even though the objective function is highly non-convex. The key theoretical tool is the measure representation of the neural networks and a novel application of a dual certificate argument for sparse estimation on a measure space. We analyze the global minima and global convergence property in the measure space.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

35 of 54 references.

References · 35

  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
23 more
  1. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  2. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  3. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  4. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  5. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  6. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  7. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  8. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  9. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  10. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  11. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  12. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  13. 2020Learning a Single Neuron with Gradient Methodscited by 50
  14. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  15. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  16. 2018Global Non-convex Optimization with Discretized Diffusionscited by 44
  17. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  18. 2016Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networkscited by 26
  19. 2013Exact Support Recovery for Sparse Spikes Deconvolutioncited by 22
  20. 2020Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methodscited by 9
  21. 2020On Sparsity in Overparametrised Shallow ReLU Networkscited by 8
  22. 2020Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamicscited by 5
  23. 2019On the linear convergence rates of exchange and continuous methods for total variation minimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 2; 2024: 1; 2025: 3; 2026: 2.

16 papers.

Sort:

  1. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  2. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 53
  3. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  4. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  5. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
  6. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  7. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  8. 2021Parallel Deep Neural Networks Have Zero Duality Gapcited by 9
  9. 2022Annihilation of Spurious Minima in Two-Layer ReLU Networkscited by 9
  10. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  11. 2023Variational formulations of ODE-Net as a mean-field optimal control problem and existence resultscited by 3
  12. 2025Sobolev acceleration for neural networkscited by 3
4 more
  1. 2025Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLUcited by 1
  2. 2026Hidden Gauge Controls Feature Specialization in ReLU Networkscited by 1
  3. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
  4. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0