# Optimal Neural Network Approximation for High-Dimensional Continuous Functions

arXiv: https://arxiv.org/abs/2409.02363 · HTML: https://mlsanity.com/paper/2409.02363/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [Neural Network Approximation: Three Hidden Layers Are Enough](https://mlsanity.com/paper/2010.14075/) (2020)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions](https://mlsanity.com/paper/1807.00297/) (2018)
- [Deep, Skinny Neural Networks are not Universal Approximators](https://mlsanity.com/paper/1810.00393/) (2018)
- [Elementary superexpressive activations](https://mlsanity.com/paper/2102.10911/) (2021)
- [Achieve the Minimum Width of Neural Networks for Universal Approximation](https://mlsanity.com/paper/2209.11395/) (2022)
- [Minimum width for universal approximation using ReLU networks on compact domain](https://mlsanity.com/paper/2309.10402/) (2023)
- [Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation](https://mlsanity.com/paper/2305.18460/) (2023)
- [Don't Fear Peculiar Activation Functions: EUAF and Beyond](https://mlsanity.com/paper/2407.09580/) (2024)

## Built on by (most cited first, 2 of 2)

- [Multi-Scale Finite Expression Method for PDEs with Oscillatory Solutions on Complex Domains](https://mlsanity.com/paper/2510.22497/) (2025)
- [Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression](https://mlsanity.com/paper/2609.25710/) (2026)

## Abstract (arXiv)

Recently, the authors of citeSYZ22 developed a neural network with width 36d(2d + 1) and depth 11, which utilizes a special activation function called the elementary universal activation function, to achieve the super approximation property for functions in C([a,b]d). That is, the constructed network only requires a fixed number of neurons (and thus parameters) to approximate a d-variate continuous function on a d-dimensional hypercube with arbitrary accuracy. More specifically, only O(d2) neurons or parameters are used. One natural question is whether we can reduce the number of these neurons or parameters in such a network. By leveraging a variant of the Kolmogorov Superposition Theorem, textcolorblackwe show that there is a composition of networks generated by the elementary universal activation function with at most 10889d + 10887 nonzero parameters such that this super approximation property is attained. The composed network consists of repeated evaluations of two neural networks: one with width 36(2d+1) and the other with width 36, both having 5 layers. Furthermore, we present a family of continuous functions that requires at least width d, and thus at least d neurons or parameters, to achieve arbitrary accuracy in its approximation. This suggests that the number of nonzero parameters is optimal in the sense that it grows linearly with the input dimension d, unlike some approximation methods where parameters may grow exponentially with d.