# A closer look at the approximation capabilities of neural networks

arXiv: https://arxiv.org/abs/2002.06505 · HTML: https://mlsanity.com/paper/2002.06505/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2020.

## Built on by (most cited first, 8 of 8)

- [Learning Hard Optimization Problems: A Data Generation Perspective](https://mlsanity.com/paper/2106.02601/) (2021)
- [Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation](https://mlsanity.com/paper/2305.18460/) (2023)
- [Abstract Visual Reasoning: An Algebraic Approach for Solving Raven's Progressive Matrices](https://mlsanity.com/paper/2303.11730/) (2023)
- [Invariant polynomials and machine learning](https://mlsanity.com/paper/2104.12733/) (2021)
- [Simulating Weighted Automata over Sequences and Trees with Transformers](https://mlsanity.com/paper/2403.09728/) (2024)
- [Universally Rank Consistent Ordinal Regression in Neural Networks](https://mlsanity.com/paper/2110.07470/) (2021)
- [Reward-Based 1-bit Compressed Federated Distillation on Blockchain](https://mlsanity.com/paper/2106.14265/) (2021)
- [Compiling to linear neurons](https://mlsanity.com/paper/2511.13769/) (2025)

## Abstract (arXiv)

The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions σ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold varepsilon, if and only if σ is non-polynomial. In this paper, we give a direct algebraic proof of the theorem. Furthermore we shall explicitly quantify the number of hidden units required for approximation. Specifically, if Xsubseteq ℝn is compact, then a neural network with n input units, m output units, and a single hidden layer with binomn+dd hidden units (independent of m and varepsilon), can uniformly approximate any polynomial function f:X → ℝm whose total degree is at most d for each of its m coordinate functions. In the general case that f is any continuous function, we show there exists some N∈ O(varepsilon-n) (independent of m), such that N hidden units would suffice to approximate f. We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights. We highlight several consequences: (i) For any δ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < δ. (ii) There exists some λ>0 (depending only on f and σ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w|>λ. (iii) If the non-bias weights in the first layer are fixed and randomly chosen from a suitable range, then the UAP holds with probability 1.