mlsanityData to 14 Oct 2026

Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs

arXiv 2306.06479 ↗First posted 10 Jun 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters. By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets. Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm. Finally we perform a range of numerical experiments, which corroborate our theoretical findings.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

42 of 51 references.

References · 42

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2017Exploring Generalization in Deep Learningcited by 600
  3. 2017Implicit Regularization in Matrix Factorizationcited by 322
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  6. 2020Directional convergence and alignment in deep learningcited by 178
  7. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  8. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  9. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  10. 2018Stochastic subgradient method converges on tame functionscited by 124
  11. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  12. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
30 more
  1. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  2. 2019How do infinite width bounded norm networks look in function space?cited by 106
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  4. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  5. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  6. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  7. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  8. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  9. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  10. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  11. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  12. 2020Learning a Single Neuron with Gradient Methodscited by 50
  13. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  14. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  15. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  16. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  17. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  18. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  19. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  20. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  21. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  22. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  23. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  24. 2023Penalising the biases in norm regularisation enforces sparsitycited by 19
  25. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  26. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  27. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  28. 2022Adversarial Reprogramming Revisitedcited by 10
  29. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  30. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 2.

16 papers.

Sort:

  1. 2025An overview of condensation phenomenon in deep learningcited by 17
  2. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  3. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  4. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  5. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  6. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  7. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  8. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 4
  9. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4
  10. 2025Building Bridges between Regression, Clustering, and Classificationcited by 2
  11. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  12. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
4 more
  1. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 1
  2. 2026When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networkscited by 1
  3. 2024Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Targetcited by 0
  4. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0