Data to 7 Oct 2026

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

arXiv 1901.08584 ↗First posted 24 Jan 2019cs.LG cs.NE stat.MLcited by 562

Not on the map. Why?

Abstract (arXiv)

Recent works have cast some light on the mystery of why deep nets fit any data and generalize despite being very overparametrized. This paper analyzes training and generalization for a simple 2-layer ReLU net with random initialization, and provides the following improvements over recent works: (i) Using a tighter characterization of training speed than recent papers, an explanation for why training a neural net with random labels leads to slower training, as originally observed in [Zhang et al. ICLR'17]. (ii) Generalization bound independent of network size, using a data-dependent complexity measure. Our measure distinguishes clearly between random labels and true labels on MNIST and CIFAR, as shown by experiments. Moreover, recent papers require sample complexity to increase (slowly) with the size, while our sample complexity is completely independent of the network size. (iii) Learnability of a broad class of smooth functions by 2-layer ReLU nets trained via gradient descent. The key idea is to track dynamics of training and generalization via properties of a related kernel.

Key facts. Cited by 562 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 137; 2022: 132; 2023: 105; 2024: 90; 2025: 64; 2026: 33.

562 papers.

Sort:

By area:

2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
2023TRAK: Attributing Model Behavior at Scalecited by 273
2021The Values Encoded in Machine Learning Researchcited by 172
2022Neural Networks can Learn Representations with Gradient Descentcited by 161
2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 147
2022If Influence Functions are the Answer, Then What is the Question?cited by 142
2021Layered Neural Atlases for Consistent Video Editingcited by 137
2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
2022A Kernel-Based View of Language Model Fine-Tuningcited by 118

38 more

2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
2021Learning with invariances in random features and kernel modelscited by 89
2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
2023Theory on Forgetting and Generalization of Continual Learningcited by 69
2022Overview frequency principle/spectral bias in deep learningcited by 68
2021Stochastic Training is Not Necessary for Generalizationcited by 63
2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
2022Fast Finite Width Neural Tangent Kernelcited by 62
2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
2021On the Frequency Bias of Generative Modelscited by 60
2022Theory of Graph Neural Networks: Representation and Learningcited by 60
2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
2022Mechanistic Mode Connectivitycited by 50
2022Learning threshold neurons via the "edge of stability"cited by 50
2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45

50 most cited of 562 All 562 (JSON)