# Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

arXiv: https://arxiv.org/abs/2409.16697 · HTML: https://mlsanity.com/paper/2409.16697/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 7 of 7)

- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)
- [Towards Understanding the Condensation of Neural Networks at Initial Training](https://mlsanity.com/paper/2105.11686/) (2021)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Approximation capability of two hidden layer feedforward neural networks with fixed weights](https://mlsanity.com/paper/2101.09181/) (2021)
- [On the approximation by single hidden layer feedforward neural networks with fixed weights](https://mlsanity.com/paper/1708.06219/) (2017)
- [A study on effectiveness of extreme learning machine](https://mlsanity.com/paper/1409.3924/) (2014)
- [Measure theoretic results for approximation by neural networks with limited weights](https://mlsanity.com/paper/2304.01880/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Neural Difference-of-Entropies Estimator for Mutual Information](https://mlsanity.com/paper/2502.13085/) (2025)

## Abstract (arXiv)

The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured? Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of textitε outer measure and textitNumerical Span Dimension (NSdim) to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.