Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons
Not on the map. Why?
Abstract (arXiv)
The minimal number of neurons required for a feedforward neural network to interpolate n generic input-output pairs from ℝd× ℝd' is Theta(√nd'). While previous results have shown that Theta(√nd') neurons are sufficient, they have been limited to sigmoid, Heaviside, and rectified linear unit (ReLU) as the activation function. Using a different approach, we prove that Theta(√nd') neurons are sufficient as long as the activation function is real analytic at a point and not a polynomial there. Thus, the only practical activation functions that our result does not apply to are piecewise polynomials. Importantly, this means that activation functions can be freely chosen in a problem-dependent manner without loss of interpolation power.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 7 of its 23 references that are in the map. First posted on arXiv 22 May 2024.
Builds on
7 of 23 references.
References · 7
- 2017Attention Is All You Needcited by 60,361
- 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
- 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 78
- 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
- 2020Provable Memorization via Deep Neural Networks using Sub-linear Parameterscited by 34
- 2021An Exponential Improvement on the Memorization Capacity of Deep Threshold Networkscited by 14
- 2023Memory capacity of two layer neural networks with smooth activationscited by 5
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.