# Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks

arXiv: https://arxiv.org/abs/2003.12895 · HTML: https://mlsanity.com/paper/2003.12895/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2020.

## Built on by (most cited first, 7 of 7)

- [On the Optimal Memorization Power of ReLU Neural Networks](https://mlsanity.com/paper/2110.03187/) (2021)
- [Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization](https://mlsanity.com/paper/2205.10217/) (2022)
- [Mixture of Parrots: Experts improve memorization more than reasoning](https://mlsanity.com/paper/2410.19034/) (2024)
- [Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularization](https://mlsanity.com/paper/2311.14609/) (2023)
- [Deep Networks Learn Deep Hierarchical Models](https://mlsanity.com/paper/2601.00455/) (2026)
- [Generalizability of Memorization Neural Networks](https://mlsanity.com/paper/2411.00372/) (2024)
- [MoRE: Scaling mixture of experts with hardware-aware low-rank routing](https://mlsanity.com/paper/2609.36301/) (2026)

## Abstract (arXiv)

We prove that a single step of gradient decent over depth two network, with q hidden neurons, starting from orthogonal initialization, can memorize Omegaleft(fracdqlog4(d)right) independent and randomly labeled Gaussians in ℝd. The result is valid for a large class of activation functions, which includes the absolute value.