# Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

arXiv: https://arxiv.org/abs/2305.18460 · HTML: https://mlsanity.com/paper/2305.18460/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 19 of 19)

- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Approximating Continuous Functions by ReLU Nets of Minimal Width](https://mlsanity.com/paper/1710.11278/) (2017)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators](https://mlsanity.com/paper/2006.11469/) (2020)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [Depth Separation for Neural Networks](https://mlsanity.com/paper/1702.08489/) (2017)
- [Deep, Skinny Neural Networks are not Universal Approximators](https://mlsanity.com/paper/1810.00393/) (2018)
- [Universal Approximation Property of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2012.02414/) (2020)
- [Achieve the Minimum Width of Neural Networks for Universal Approximation](https://mlsanity.com/paper/2209.11395/) (2022)
- [Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regions](https://mlsanity.com/paper/1803.00094/) (2018)
- [A closer look at the approximation capabilities of neural networks](https://mlsanity.com/paper/2002.06505/) (2020)
- [Universal Approximation of Residual Flows in Maximum Mean Discrepancy](https://mlsanity.com/paper/2103.05793/) (2021)
- [On decision regions of narrow deep neural networks](https://mlsanity.com/paper/1807.01194/) (2018)
- [Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems](https://mlsanity.com/paper/2209.10909/) (2022)
- [Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units](https://mlsanity.com/paper/1303.7461/) (2013)
- [Residual Networks as Geodesic Flows of Diffeomorphisms](https://mlsanity.com/paper/1805.09585/) (2018)

## Built on by (most cited first, 13 of 13)

- [LoRA Training in the NTK Regime has No Spurious Local Minima](https://mlsanity.com/paper/2402.11867/) (2024)
- [Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning](https://mlsanity.com/paper/2411.00835/) (2024)
- [A Minimal Control Family of Dynamical Systems for Universal Approximation](https://mlsanity.com/paper/2312.12903/) (2023)
- [Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines](https://mlsanity.com/paper/2405.13682/) (2024)
- [Optimal Neural Network Approximation for High-Dimensional Continuous Functions](https://mlsanity.com/paper/2409.02363/) (2024)
- [Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networks](https://mlsanity.com/paper/2409.06555/) (2024)
- [A holomorphic neural network framework for 3D boundary value problems governed by harmonic potentials](https://mlsanity.com/paper/2605.31231/) (2026)
- [Low-dimensional topology of deep neural networks](https://mlsanity.com/paper/2606.31856/) (2026)
- [AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery](https://mlsanity.com/paper/2504.01875/) (2025)
- [Neural Network-assisted Interval Reachability for Systems with Control Barrier Function-Based Safe Controllers](https://mlsanity.com/paper/2504.08249/) (2025)
- [Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding](https://mlsanity.com/paper/2511.06376/) (2025)
- [Approximation Theory for Neural Networks: Old and New](https://mlsanity.com/paper/2605.21451/) (2026)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)

## Abstract (arXiv)

The study of universal approximation properties (UAP) for neural networks (NN) has a long history. When the network width is unlimited, only a single hidden layer is sufficient for UAP. In contrast, when the depth is unlimited, the width for UAP needs to be not less than the critical width w^*min=max(dx,dy), where dx and dy are the dimensions of the input and output, respectively. Recently, citecai2022achieve shows that a leaky-ReLU NN with this critical width can achieve UAP for Lp functions on a compact domain K, i.e., the UAP for Lp(K,ℝdy). This paper examines a uniform UAP for the function class C(K,ℝdy) and gives the exact minimum width of the leaky-ReLU NN as wmin=max(dx,dy)+Delta (dx, dy), where Delta (dx, dy) is the additional dimensions for approximating continuous functions with diffeomorphisms via embedding. To obtain this result, we propose a novel lift-flow-discretization approach that shows that the uniform UAP has a deep connection with topological theory.