# Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently

arXiv: https://arxiv.org/abs/1909.11837 · HTML: https://mlsanity.com/paper/1909.11837/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization](https://mlsanity.com/paper/2205.10217/) (2022)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [Fixed width treelike neural networks capacity analysis -- generic activations](https://mlsanity.com/paper/2402.05696/) (2024)
- [Capacity of the treelike sign perceptrons neural networks with one hidden layer -- RDT based upper bounds](https://mlsanity.com/paper/2312.08244/) (2023)
- [Lifted RDT based capacity analysis of the 1-hidden layer treelike sign perceptrons neural networks](https://mlsanity.com/paper/2312.08257/) (2023)
- [Memorization With Neural Nets: Going Beyond the Worst Case](https://mlsanity.com/paper/2310.00327/) (2023)
- [Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/2103.01887/) (2021)

## Abstract (arXiv)

It has been observed citepzhang2016understanding that deep neural networks can memorize: they achieve 100% accuracy on training data. Recent theoretical results explained such behavior in highly overparametrized regimes, where the number of neurons in each layer is larger than the number of training samples. In this paper, we show that neural networks can be trained to memorize training data perfectly in a mildly overparametrized regime, where the number of parameters is just a constant factor more than the number of training samples, and the number of neurons is much smaller.