# Achieve the Minimum Width of Neural Networks for Universal Approximation

arXiv: https://arxiv.org/abs/2209.11395 · HTML: https://mlsanity.com/paper/2209.11395/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [Deep, Skinny Neural Networks are not Universal Approximators](https://mlsanity.com/paper/1810.00393/) (2018)
- [Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory](https://mlsanity.com/paper/2007.06007/) (2020)
- [Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems](https://mlsanity.com/paper/2209.10909/) (2022)

## Built on by (most cited first, 26 of 26)

- [Minimum width for universal approximation using ReLU networks on compact domain](https://mlsanity.com/paper/2309.10402/) (2023)
- [Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation](https://mlsanity.com/paper/2305.18460/) (2023)
- [Addressing common misinterpretations of KART and UAT in neural network literature](https://mlsanity.com/paper/2408.16389/) (2024)
- [Improved weight initialization for deep and narrow feedforward neural network](https://mlsanity.com/paper/2311.03733/) (2023)
- [Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions](https://mlsanity.com/paper/2305.12205/) (2023)
- [Universal approximation with complex-valued deep narrow neural networks](https://mlsanity.com/paper/2305.16910/) (2023)
- [A Minimal Control Family of Dynamical Systems for Universal Approximation](https://mlsanity.com/paper/2312.12903/) (2023)
- [Minimum number of neurons in fully connected layers of a given neural network (the first approximation)](https://mlsanity.com/paper/2405.14147/) (2024)
- [Minimum width for universal approximation using squashable activation functions](https://mlsanity.com/paper/2504.07371/) (2025)
- [Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks](https://mlsanity.com/paper/2310.03529/) (2023)
- [Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines](https://mlsanity.com/paper/2405.13682/) (2024)
- [Optimal Neural Network Approximation for High-Dimensional Continuous Functions](https://mlsanity.com/paper/2409.02363/) (2024)
- [Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networks](https://mlsanity.com/paper/2409.06555/) (2024)
- [Approximation properties of neural ODEs](https://mlsanity.com/paper/2503.15696/) (2025)
- [Minimum Width of Deep Narrow Networks for Universal Approximation](https://mlsanity.com/paper/2511.06837/) (2025)
- [A holomorphic neural network framework for 3D boundary value problems governed by harmonic potentials](https://mlsanity.com/paper/2605.31231/) (2026)
- [Neural Networks Trained by Weight Permutation are Universal Approximators](https://mlsanity.com/paper/2407.01033/) (2024)
- [Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regime](https://mlsanity.com/paper/2506.24120/) (2025)
- [Low-dimensional topology of deep neural networks](https://mlsanity.com/paper/2606.31856/) (2026)
- [2-Cats: 2D Copula Approximating Transforms](https://mlsanity.com/paper/2309.16391/) (2023)
- [Achieving interpretable machine learning by functional decomposition of black-box models into explainable predictor effects](https://mlsanity.com/paper/2407.18650/) (2024)
- [Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding](https://mlsanity.com/paper/2511.06376/) (2025)
- [Approximation Theory for Neural Networks: Old and New](https://mlsanity.com/paper/2605.21451/) (2026)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)
- [Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One](https://mlsanity.com/paper/2607.04597/) (2026)
- [The minimal width of neural networks and immersions of real projective space](https://mlsanity.com/paper/2610.20229/) (2026)

## Abstract (arXiv)

The universal approximation property (UAP) of neural networks is fundamental for deep learning, and it is well known that wide neural networks are universal approximators of continuous functions within both the Lp norm and the continuous/uniform norm. However, the exact minimum width, wmin, for the UAP has not been studied thoroughly. Recently, using a decoder-memorizer-encoder scheme, citetPark2021Minimum found that wmin = max(dx+1,dy) for both the Lp-UAP of ReLU networks and the C-UAP of ReLU+STEP networks, where dx,dy are the input and output dimensions, respectively. In this paper, we consider neural networks with an arbitrary set of activation functions. We prove that both C-UAP and Lp-UAP for functions on compact domains share a universal lower bound of the minimal width; that is, w^*min = max(dx,dy). In particular, the critical width, w^*min, for Lp-UAP can be achieved by leaky-ReLU networks, provided that the input or output dimension is larger than one. Our construction is based on the approximation power of neural ordinary differential equations and the ability to approximate flow maps by neural networks. The nonmonotone or discontinuous activation functions case and the one-dimensional case are also discussed.