mlsanityData to 7 Oct 2026

Learning One-hidden-layer ReLU Networks via Gradient Descent

arXiv 1806.07808 ↗First posted 20 Jun 2018stat.ML cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher network. We analyze the performance of gradient descent for training such kind of neural networks based on empirical risk minimization, and provide algorithm-dependent guarantees. In particular, we prove that tensor initialization followed by gradient descent can converge to the ground-truth parameters at a linear rate up to some statistical error. To the best of our knowledge, this is the first work characterizing the recovery guarantee for practical learning of one-hidden-layer ReLU networks with multiple neurons. Numerical experiments verify our theoretical findings.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 13; 2023: 12; 2024: 8; 2025: 2; 2026: 3.

50 papers.

Sort:

By area:

  1. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  2. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 82
  3. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  4. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  5. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  6. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  7. 2022Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networkscited by 27
  8. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  9. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  10. 2024Provably learning a multi-head attention layercited by 21
  11. 2022How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysiscited by 18
  12. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
38 more
  1. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15
  2. 2023Learning Narrow One-Hidden-Layer ReLU Networkscited by 14
  3. 2022Local Identifiability of Deep ReLU Neural Networks: the Theorycited by 12
  4. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  5. 2024Towards Robust Out-of-Distribution Generalization Bounds via Sharpnesscited by 10
  6. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  7. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  8. 2023A faster and simpler algorithm for learning shallow networkscited by 7
  9. 2021Efficiently Learning Any One Hidden Layer ReLU Network From Queriescited by 6
  10. 2024How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalancecited by 6
  11. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  12. 2022Finite Sample Identification of Wide Shallow Neural Networks with Biasescited by 5
  13. 2023Continuized Acceleration for Quasar Convex Functions in Non-Convex Optimizationcited by 5
  14. 2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4
  15. 2021How much pre-training is enough to discover a good subnetwork?cited by 4
  16. 2022Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysiscited by 4
  17. 2023Toward L_∞-recovery of Nonlinear Functions: A Polynomial Sample Complexity Bound for Gaussian Random Fieldscited by 4
  18. 2022Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recoverycited by 3
  19. 2023Max-affine regression via first-order methodscited by 3
  20. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  21. 2025Sobolev acceleration for neural networkscited by 3
  22. 2021Towards Searching Efficient and Accurate Neural Network Architectures in Binary Classification Problemscited by 2
  23. 2021Coordinate Descent Methods for DC Minimization: Optimality Conditions and Global Convergencecited by 2
  24. 2023Most Neural Networks Are Almost Learnablecited by 2
  25. 2024Understanding Representation Learnability of Nonlinear Self-Supervised Learningcited by 2
  26. 2024Stochastic Bandits with ReLU Neural Networkscited by 2
  27. 2021Structured Directional Pruning via Perturbation Orthogonal Projectioncited by 1
  28. 2022Explicitising The Implicit Intrepretability of Deep Neural Networks Via Dualitycited by 1
  29. 2023Convergence Analysis for Learning Orthonormal Deep Linear Neural Networkscited by 1
  30. 2024On the Hardness of Learning One Hidden Layer Neural Networkscited by 1
  31. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Pointcited by 1
  32. 2026When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networkscited by 1
  33. 2021Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networkscited by 0
  34. 2022Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensionscited by 0
  35. 2023Improving the Expressive Power of Deep Neural Networks through Integral Activation Transformcited by 0
  36. 2024Iterative thresholding for non-linear learning in the strong varepsilon-contamination modelcited by 0
  37. 2026Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Datacited by 0
  38. 2026Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networkscited by 0