# Minimum Width for Universal Approximation

arXiv: https://arxiv.org/abs/2006.08859 · HTML: https://mlsanity.com/paper/2006.08859/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 30 of 94)

- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees](https://mlsanity.com/paper/2410.10807/) (2024)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [Achieve the Minimum Width of Neural Networks for Universal Approximation](https://mlsanity.com/paper/2209.11395/) (2022)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024)
- [Minimum width for universal approximation using ReLU networks on compact domain](https://mlsanity.com/paper/2309.10402/) (2023)
- [Visualizing high-dimensional loss landscapes with Hessian directions](https://mlsanity.com/paper/2208.13219/) (2022)
- [A Survey on Universal Approximation Theorems](https://mlsanity.com/paper/2407.12895/) (2024)
- [Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation](https://mlsanity.com/paper/2305.18460/) (2023)
- [A likelihood approach to nonparametric estimation of a singular distribution using deep generative models](https://mlsanity.com/paper/2105.04046/) (2021)
- [Small Transformers Compute Universal Metric Embeddings](https://mlsanity.com/paper/2209.06788/) (2022)
- [Minimal Width for Universal Property of Deep RNN](https://mlsanity.com/paper/2211.13866/) (2022)
- [Solving Empirical Bayes via Transformers](https://mlsanity.com/paper/2502.09844/) (2025)
- [Width is Less Important than Depth in ReLU Neural Networks](https://mlsanity.com/paper/2202.03841/) (2022)
- [Optimization Over Trained Neural Networks: Taking a Relaxing Walk](https://mlsanity.com/paper/2401.03451/) (2024)
- [PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos](https://mlsanity.com/paper/2404.08921/) (2024)
- [Spectrally Adapted Physics-Informed Neural Networks for Solving Unbounded Domain Problems](https://mlsanity.com/paper/2202.02710/) (2022)
- [On the Nonlinearity of Layer Normalization](https://mlsanity.com/paper/2406.01255/) (2024)
- [Addressing common misinterpretations of KART and UAT in neural network literature](https://mlsanity.com/paper/2408.16389/) (2024)
- [LU-Net: Invertible Neural Networks Based on Matrix Factorization](https://mlsanity.com/paper/2302.10524/) (2023)
- [Improved weight initialization for deep and narrow feedforward neural network](https://mlsanity.com/paper/2311.03733/) (2023)
- [Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networks](https://mlsanity.com/paper/2201.06126/) (2022)
- [Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?](https://mlsanity.com/paper/2204.11231/) (2022)
- [Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions](https://mlsanity.com/paper/2305.12205/) (2023)

## Abstract (arXiv)

The universal approximation property of width-bounded networks has been studied as a dual of classical universal approximation results on depth-bounded networks. However, the critical width enabling the universal approximation has not been exactly characterized in terms of the input dimension dx and the output dimension dy. In this work, we provide the first definitive result in this direction for networks using the ReLU activation functions: The minimum width required for the universal approximation of the Lp functions is exactly maxdx+1,dy. We also prove that the same conclusion does not hold for the uniform approximation with ReLU, but does hold with an additional threshold activation function. Our proof technique can be also used to derive a tighter upper bound on the minimum width required for the universal approximation using networks with general activation functions.