# Network size and weights size for memorization with two-layers neural networks

arXiv: https://arxiv.org/abs/2006.02855 · HTML: https://mlsanity.com/paper/2006.02855/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2020.

## Built on by (most cited first, 17 of 17)

- [On the Optimal Memorization Power of ReLU Neural Networks](https://mlsanity.com/paper/2110.03187/) (2021)
- [Memorization Capacity of Multi-Head Attention in Transformers](https://mlsanity.com/paper/2306.02010/) (2023)
- [Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization](https://mlsanity.com/paper/2205.10217/) (2022)
- [Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power](https://mlsanity.com/paper/2205.13863/) (2022)
- [An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks](https://mlsanity.com/paper/2106.07724/) (2021)
- [When Are Solutions Connected in Deep Networks?](https://mlsanity.com/paper/2102.09671/) (2021)
- [NeuFENet: Neural Finite Element Solutions with Theoretical Bounds for Parametric PDEs](https://mlsanity.com/paper/2110.01601/) (2021)
- [A Recipe for Global Convergence Guarantee in Deep Neural Networks](https://mlsanity.com/paper/2104.05785/) (2021)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [Knowledge Infused Policy Gradients for Adaptive Pandemic Control](https://mlsanity.com/paper/2102.06245/) (2021)
- [What do larger image classifiers memorise?](https://mlsanity.com/paper/2310.05337/) (2023)
- [Finite Sample Identification of Wide Shallow Neural Networks with Biases](https://mlsanity.com/paper/2211.04589/) (2022)
- [Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularization](https://mlsanity.com/paper/2311.14609/) (2023)
- [Memorization With Neural Nets: Going Beyond the Worst Case](https://mlsanity.com/paper/2310.00327/) (2023)
- [MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers](https://mlsanity.com/paper/2607.10034/) (2026)
- [Neural Networks with Complex-Valued Weights Have No Spurious Local Minima](https://mlsanity.com/paper/2103.07287/) (2021)
- [Memorization capacity of deep ReLU neural networks characterized by width and depth](https://mlsanity.com/paper/2603.09589/) (2026)

## Abstract (arXiv)

In 1988, Eric B. Baum showed that two-layers neural networks with threshold activation function can perfectly memorize the binary labels of n points in general position in ℝd using only ulcorner n/d urcorner neurons. We observe that with ReLU networks, using four times as many neurons one can fit arbitrary real labels. Moreover, for approximate memorization up to error ε, the neural tangent kernel can also memorize with only Oleft(fracnd · log(1/ε) right) neurons (assuming that the data is well dispersed too). We show however that these constructions give rise to networks where the magnitude of the neurons' weights are far from optimal. In contrast we propose a new training procedure for ReLU networks, based on complex (as opposed to real) recombination of the neurons, for which we show approximate memorization with both Oleft(fracnd · fraclog(1/ε)εright) neurons, as well as nearly-optimal size of the weights.