Memorizing Gaussians with no over-parameterizaion via gradient decent on neural networks
Not on the map. Why?
Abstract (arXiv)
We prove that a single step of gradient decent over depth two network, with q hidden neurons, starting from orthogonal initialization, can memorize Omegaleft(fracdqlog4(d)right) independent and randomly labeled Gaussians in ℝd. The result is valid for a large class of activation functions, which includes the absolute value.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 2; 2025: 0; 2026: 2.
7 papers.
- 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
- 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
- 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
- 2023Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularizationcited by 4
- 2026Deep Networks Learn Deep Hierarchical Modelscited by 2
- 2024Generalizability of Memorization Neural Networkscited by 1
- 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
- 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing28 Sep 2026
- 2026Deep Networks Learn Deep Hierarchical Models1 Jan 2026
- 2024Generalizability of Memorization Neural Networks1 Nov 2024
- 2024Mixture of Parrots: Experts improve memorization more than reasoning24 Oct 2024
- 2023Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularization24 Nov 2023
- 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization20 May 2022
- 2021On the Optimal Memorization Power of ReLU Neural Networks7 Oct 2021