# Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons

arXiv: https://arxiv.org/abs/2405.13738 · HTML: https://mlsanity.com/paper/2405.13738/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks](https://mlsanity.com/paper/2012.11654/) (2020)
- [Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity](https://mlsanity.com/paper/1810.07770/) (2018)
- [Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization](https://mlsanity.com/paper/2205.10217/) (2022)
- [Provable Memorization via Deep Neural Networks using Sub-linear Parameters](https://mlsanity.com/paper/2010.13363/) (2020)
- [An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks](https://mlsanity.com/paper/2106.07724/) (2021)
- [Memory capacity of two layer neural networks with smooth activations](https://mlsanity.com/paper/2308.02001/) (2023)

## Built on by (most cited first, 2 of 2)

- [Memorization Capacity for Additive Fine-Tuning with Small ReLU Networks](https://mlsanity.com/paper/2408.00359/) (2024)
- [Taming the Loss Landscape of PINNs with Noisy Feynman-Kac Supervision: Operator Preconditioning and Non-Asymptotic Error Bounds](https://mlsanity.com/paper/2606.00643/) (2026)

## Abstract (arXiv)

The minimal number of neurons required for a feedforward neural network to interpolate n generic input-output pairs from ℝd× ℝd' is Theta(√nd'). While previous results have shown that Theta(√nd') neurons are sufficient, they have been limited to sigmoid, Heaviside, and rectified linear unit (ReLU) as the activation function. Using a different approach, we prove that Theta(√nd') neurons are sufficient as long as the activation function is real analytic at a point and not a polynomial there. Thus, the only practical activation functions that our result does not apply to are piecewise polynomials. Importantly, this means that activation functions can be freely chosen in a problem-dependent manner without loss of interpolation power.