# Mathematical Models of Overparameterized Neural Networks

arXiv: https://arxiv.org/abs/2012.13982 · HTML: https://mlsanity.com/paper/2012.13982/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2020.

## Built on by (most cited first, 15 of 15)

- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Local Augmentation for Graph Neural Networks](https://mlsanity.com/paper/2109.03856/) (2021)
- [Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity](https://mlsanity.com/paper/2202.12482/) (2022)
- [Restricted Strong Convexity of Deep Learning Models with Smooth Activations](https://mlsanity.com/paper/2209.15106/) (2022)
- [Eight challenges in developing theory of intelligence](https://mlsanity.com/paper/2306.11232/) (2023)
- [Fast Conditional Network Compression Using Bayesian HyperNetworks](https://mlsanity.com/paper/2205.06404/) (2022)
- [Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration](https://mlsanity.com/paper/2409.18461/) (2024)
- [Generalization Error of Graph Neural Networks in the Mean-field Regime](https://mlsanity.com/paper/2402.07025/) (2024)
- [Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems](https://mlsanity.com/paper/2309.12128/) (2023)
- [Generalization of GANs and overparameterized models under Lipschitz continuity](https://mlsanity.com/paper/2104.02388/) (2021)
- [Convergence Guarantees of Overparametrized Wide Deep Inverse Prior](https://mlsanity.com/paper/2303.11265/) (2023)
- [Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems trained with Gradient Descent](https://mlsanity.com/paper/2403.05395/) (2024)
- [Understanding Transfer Learning via Mean-field Analysis](https://mlsanity.com/paper/2410.17128/) (2024)
- [Implicit Regularization of the Deep Inverse Prior Trained with Inertia](https://mlsanity.com/paper/2506.02986/) (2025)

## Abstract (arXiv)

Deep learning has received considerable empirical successes in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there has been a lack of theoretical understanding for tricks invented in the deep learning literature. Known by practitioners that overparameterized neural networks are easy to learn, in the past few years there have been important theoretical developments in the analysis of overparameterized neural networks. In particular, it was shown that such systems behave like convex systems under various restricted settings, such as for two-layer NNs, and when learning is restricted locally in the so-called neural tangent kernel space around specialized initializations. This paper discusses some of these recent progresses leading to significant better understanding of neural networks. We will focus on the analysis of two-layer neural networks, and explain the key mathematical models, with their algorithmic implications. We will then discuss challenges in understanding deep neural networks and some current research directions.