Data to 14 Oct 2026

Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks

arXiv 1805.12076 ↗First posted 30 May 2018cs.LG stat.MLcited by 142

Not on the map. Why?

Abstract (arXiv)

Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes, and could potentially explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

Key facts. Cited by 142 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 46; 2022: 29; 2023: 20; 2024: 20; 2025: 17; 2026: 9.

142 papers.

Sort:

By area:

2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
2021Theory of overparametrization in quantum neural networkscited by 100
2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
2021Stochastic Training is Not Necessary for Generalizationcited by 63
2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 40
2021Understanding the role of importance weighting for deep learningcited by 39
2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 37

38 more

2025Symmetry in Neural Network Parameter Spacescited by 36
2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
2024Zero-shot forecasting of chaotic systemscited by 28
2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 26
2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
2021Measuring Generalization with Optimal Transportcited by 25
2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
2022On the Generalization Mystery in Deep Learningcited by 25
2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 21
2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
2022UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographycited by 19
2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
2021Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLPcited by 17
2021An Exponential Improvement on the Memorization Capacity of Deep Threshold Networkscited by 14
2021Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsitycited by 14
2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
2022Combined Federated and Split Learning in Edge Computing for Ubiquitous Intelligence in Internet of Things: State of the Art and Future Directionscited by 14
2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13
2021On the interplay between data structure and loss function in classification problemscited by 13
2021An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximationcited by 13
2021Efficient and Private Federated Learning with Partially Trainable Networkscited by 13
2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
2022The activity-weight duality in feed forward neural networks: The geometric determinants of generalizationcited by 12
2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
2021Training Over-parameterized Models with Non-decomposable Objectivescited by 11
2024Residual Connections Harm Generative Representation Learningcited by 11
2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 11
2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
2022Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processingcited by 10
2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10

50 most cited of 142 All 142 (JSON)