mlsanityData to 7 Oct 2026

A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

arXiv 1904.04326 ↗First posted 8 Apr 2019cs.LG math.OC stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 11; 2023: 11; 2024: 5; 2025: 4; 2026: 3.

49 papers.

Sort:

By area:

  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  3. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  4. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  5. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  6. 2022Convergence of gradient descent for deep neural networkscited by 24
  7. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  8. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  9. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  10. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  11. 2021Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networkscited by 14
  12. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
37 more
  1. 2021Generalization Guarantees for Neural Architecture Search with Train-Validation Splitcited by 14
  2. 2021An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximationcited by 13
  3. 2021Implicit Bias of Linear RNNscited by 11
  4. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  5. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  6. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  7. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  8. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  9. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  10. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  11. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  12. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  13. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  14. 2023Learning Hierarchical Polynomials with Three-Layer Neural Networkscited by 8
  15. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 7
  16. 2024Nesterov acceleration in benignly non-convex landscapescited by 6
  17. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  18. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  19. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  20. 2022SPADE4: Sparsity and Delay Embedding based Forecasting of Epidemicscited by 4
  21. 2023Reinforcement Learning with Function Approximation: From Linear to Nonlinearcited by 4
  22. 2022A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networkscited by 3
  23. 2023A qualitative difference between gradient flows of convex functions in finite- and infinite-dimensional Hilbert spacescited by 3
  24. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  25. 2024Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Featurescited by 3
  26. 2025Orthogonal greedy algorithm for linear operator learning with shallow neural networkcited by 3
  27. 2021Nonasymptotic theory for two-layer neural networks: Beyond the bias-variance trade-offcited by 2
  28. 2025Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variablescited by 2
  29. 2024Deep Learning without Global Optimization by Random Fourier Neural Networkscited by 1
  30. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  31. 2022Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provablycited by 0
  32. 2023The L^∞ Learnability of Reproducing Kernel Hilbert Spacescited by 0
  33. 2023Gibbs-Based Information Criteria and the Over-Parameterized Regimecited by 0
  34. 2024Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Trainingcited by 0
  35. 2026A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learningcited by 0
  36. 2026Deciphering Neural Reparameterized Full-Waveform Inversion with Neural Sensitivity Kernel and Wave Tangent Kernelcited by 0
  37. 2026The double descent and Runge phenomena in overparametrized polynomial interpolationcited by 0