mlsanityData to 7 Oct 2026

Learning ReLUs via Gradient Descent

arXiv 1705.04591 ↗First posted 10 May 2017cs.LG cs.IT math.ITcited by 54

Not on the map. Why?

Abstract (arXiv)

In this paper we study the problem of learning Rectified Linear Units (ReLUs) which are functions of the form max(0,<w,x>) with w denoting the weight vector. We study this problem in the high-dimensional regime where the number of observations are fewer than the dimension of the weight vector. We assume that the weight vector belongs to some closed set (convex or nonconvex) which captures known side-information about its structure. We focus on the realizable model where the inputs are chosen i.i.d.~from a Gaussian distribution and the labels are generated according to a planted weight vector. We show that projected gradient descent, when initialization at 0, converges at a linear rate to the planted model with a number of samples that is optimal up to numerical constants. Our results on the dynamics of convergence of these very shallow neural nets may provide some insights towards understanding the dynamics of deeper architectures.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 13; 2023: 16; 2024: 9; 2025: 3; 2026: 5.

54 papers.

Sort:

  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  3. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  4. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  5. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  6. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
  7. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  8. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  9. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  10. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  11. 2023Near-Optimal Cryptographic Hardness of Agnostically Learning Halfspaces and ReLU Regression under Gaussian Marginalscited by 31
  12. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
38 more
  1. 2024Provably learning a multi-head attention layercited by 21
  2. 2021On the Cryptographic Hardness of Learning Single Periodic Neuronscited by 20
  3. 2022Learning a Single Neuron with Adversarial Label Noise via Gradient Descentcited by 20
  4. 2021GradSign: Model Performance Inference with Theoretical Insightscited by 18
  5. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  6. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  7. 2023On Single Index Models beyond Gaussian Datacited by 17
  8. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  9. 2023Learning Narrow One-Hidden-Layer ReLU Networkscited by 14
  10. 2022Learning a Single Neuron for Non-monotonic Activation Functionscited by 13
  11. 2022Agnostic Learning of General ReLU Activation Using Gradient Descentcited by 12
  12. 2023Robustly Learning a Single Neuron via Sharpnesscited by 12
  13. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  14. 2021ReLU Regression with Massart Noisecited by 10
  15. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  16. 2023Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuroncited by 8
  17. 2023Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network?cited by 8
  18. 2023A faster and simpler algorithm for learning shallow networkscited by 7
  19. 2022SQ Lower Bounds for Learning Single Neurons with Massart Noisecited by 5
  20. 2022Finite Sample Identification of Wide Shallow Neural Networks with Biasescited by 5
  21. 2024Agnostic Learning of Arbitrary ReLU Activation under Gaussian Marginalscited by 5
  22. 2024Symmetric Matrix Completion with ReLU Samplingcited by 4
  23. 2024Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Modelscited by 4
  24. 2021Theoretical Exploration of Flexible Transmitter Modelcited by 3
  25. 2022Overparameterized ReLU Neural Networks Learn the Simplest Models: Neural Isometry and Exact Recoverycited by 3
  26. 2023Max-affine regression via first-order methodscited by 3
  27. 2023Gradient Descent Provably Solves Nonlinear Tomographic Reconstructioncited by 3
  28. 2024Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noisecited by 2
  29. 2023Distribution-Independent Regression for Generalized Linear Models with Oblivious Corruptionscited by 1
  30. 2024Improving the Convergence Rates of Forward Gradient Descent with Repeated Samplingcited by 1
  31. 2025Joint Learning in the Gaussian Single Index Modelcited by 1
  32. 2026Interactive Learning of Single-Index Models via Stochastic Gradient Descentcited by 1
  33. 2026When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networkscited by 1
  34. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  35. 2022Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensionscited by 0
  36. 2024Inferring Change Points in High-Dimensional Regression via Approximate Message Passingcited by 0
  37. 2024On subdifferential chain rule of matrix factorization and beyondcited by 0
  38. 2025AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recoverycited by 0

50 most cited of 54 All 54 (JSON)