mlsanityData to 14 Oct 2026

Gradient Dynamics of Shallow Univariate ReLU Networks

arXiv 1906.07842 ↗First posted 18 Jun 2019cs.LG stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

We present a theoretical and empirical study of the gradient dynamics of overparameterized shallow ReLU networks with one-dimensional input, solving least-squares interpolation. We show that the gradient dynamics of such networks are determined by the gradient flow in a non-redundant parameterization of the network function. We examine the principal qualitative features of this gradient flow. In particular, we determine conditions for two learning regimes:kernel and adaptive, which depend both on the relative magnitude of initialization of weights in different layers and the asymptotic behavior of initialization coefficients in the limit of large network widths. We show that learning in the kernel regime yields smooth interpolants, minimizing curvature, and reduces to cubic splines for uniform initializations. Learning in the adaptive regime favors instead linear splines, where knots cluster adaptively at the sample points.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 10; 2023: 4; 2024: 7; 2025: 7; 2026: 4.

46 papers.

Sort:

By area:

  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  2. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  3. 2021Neural Fields as Learnable Kernels for 3D Reconstructioncited by 58
  4. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  5. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  6. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  7. 2021Geometry of Linear Convolutional Networkscited by 27
  8. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  9. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  10. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  11. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  12. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
34 more
  1. 2021On the Expected Complexity of Maxout Networkscited by 15
  2. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  3. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  4. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  5. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  6. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  7. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11
  8. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  9. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 10
  10. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  11. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  12. 2021On Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
  13. 2023Implicit Regularization for Group Sparsitycited by 8
  14. 2022NeuralMLS: Geometry-Aware Control Point Deformationcited by 5
  15. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  16. 2022Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficultycited by 5
  17. 2021Universal scaling laws in the gradient descent training of neural networkscited by 4
  18. 2021How Infinitely Wide Neural Networks Can Benefit from Multi-task Learning -- an Exact Macroscopic Characterizationcited by 4
  19. 2022Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergencecited by 3
  20. 2024Prices, Bids, Values: One ML-Powered Combinatorial Auction to Rule Them Allcited by 2
  21. 2021Separation Results between Fixed-Kernel and Feature-Learning Probability Metricscited by 1
  22. 2023Inferring Inferencecited by 1
  23. 2024A topological description of loss surfaces based on Betti Numberscited by 1
  24. 2025Gradient Descent as a Shrinkage Operator for Spectral Biascited by 1
  25. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  26. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 1
  27. 2026Optimal score function estimation via derivatives constraintscited by 1
  28. 2026Hidden Gauge Controls Feature Specialization in ReLU Networkscited by 1
  29. 2021Two-argument activation functions learn soft XOR operations like cortical neuronscited by 0
  30. 2023Frequency maps reveal the correlation between Adversarial Attacks and Implicit Biascited by 0
  31. 2024Demystifying Lazy Training of Neural Networks from a Macroscopic Viewpointcited by 0
  32. 2024Equidistribution-based training of Free Knot Splines and ReLU Neural Networkscited by 0
  33. 2025The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearitycited by 0
  34. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0