mlsanityData to 7 Oct 2026

Memory capacity of neural networks with threshold and ReLU activations

arXiv 2001.06938 ↗First posted 20 Jan 2020cs.LG cs.IT cs.NEcited by 13

Not on the map. Why?

Abstract (arXiv)

Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks -- those with more connections than the size of the training data -- are often able to memorize the training data with 100% accuracy. This was rigorously proved for networks with sigmoid activation functions and, very recently, for ReLU activations. Addressing a 1988 open question of Baum, we prove that this phenomenon holds for general multilayered perceptrons, i.e. neural networks with threshold activation functions, or with any mix of threshold and ReLU activations. Our construction is probabilistic and exploits sparsity.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 1; 2023: 5; 2024: 3; 2025: 0; 2026: 2.

13 papers.

Sort:

  1. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  2. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  3. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 39
  4. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  5. 2023Globally Optimal Training of Neural Networks with Threshold Activation Functionscited by 9
  6. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  7. 2024Fixed width treelike neural networks capacity analysis -- generic activationscited by 4
  8. 2023Capacity of the treelike sign perceptrons neural networks with one hidden layer -- RDT based upper boundscited by 3
  9. 2023Lifted RDT based capacity analysis of the 1-hidden layer treelike sign perceptrons neural networkscited by 3
  10. 2023Memorization With Neural Nets: Going Beyond the Worst Casecited by 1
  11. 2026Effective Frontiers: A Unification of Neural Scaling Lawscited by 1
  12. 2024Memorization Capacity for Additive Fine-Tuning with Small ReLU Networkscited by 0
1 more
  1. 2026Information Theoretic Perspective on Representation Learningcited by 0