mlsanityData to 14 Oct 2026

Achieve the Minimum Width of Neural Networks for Universal Approximation

arXiv 2209.11395 ↗First posted 23 Sep 2022cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

The universal approximation property (UAP) of neural networks is fundamental for deep learning, and it is well known that wide neural networks are universal approximators of continuous functions within both the Lp norm and the continuous/uniform norm. However, the exact minimum width, wmin, for the UAP has not been studied thoroughly. Recently, using a decoder-memorizer-encoder scheme, citetPark2021Minimum found that wmin = max(dx+1,dy) for both the Lp-UAP of ReLU networks and the C-UAP of ReLU+STEP networks, where dx,dy are the input and output dimensions, respectively. In this paper, we consider neural networks with an arbitrary set of activation functions. We prove that both C-UAP and Lp-UAP for functions on compact domains share a universal lower bound of the minimal width; that is, w^*min = max(dx,dy). In particular, the critical width, w^*min, for Lp-UAP can be achieved by leaky-ReLU networks, provided that the input or output dimension is larger than one. Our construction is based on the approximation power of neural ordinary differential equations and the ability to approximate flow maps by neural networks. The nonmonotone or discontinuous activation functions case and the one-dimensional case are also discussed.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 16 of its 31 references that are in the map. First posted on arXiv 23 Sep 2022.

Builds on

16 of 31 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  4. 2016Benefits of depth in neural networkscited by 250
  5. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  6. 2019Universal Approximation with Deep Narrow Networkscited by 200
  7. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 122
  8. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  9. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  10. 2020Minimum Width for Universal Approximationcited by 94
  11. 2019The phase diagram of approximation rates for deep neural networkscited by 88
  12. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
4 more
  1. 2021Neural ODE control for classification, approximation and transportcited by 57
  2. 2018Deep, Skinny Neural Networks are not Universal Approximatorscited by 38
  3. 2020Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theorycited by 18
  4. 2022Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systemscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 7; 2025: 5; 2026: 6.

26 papers.

Sort:

  1. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 18
  2. 2023Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximationcited by 13
  3. 2024Addressing common misinterpretations of KART and UAT in neural network literaturecited by 9
  4. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
  5. 2023Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositionscited by 6
  6. 2023Universal approximation with complex-valued deep narrow neural networkscited by 3
  7. 2023A Minimal Control Family of Dynamical Systems for Universal Approximationcited by 3
  8. 2024Minimum number of neurons in fully connected layers of a given neural network (the first approximation)cited by 3
  9. 2025Minimum width for universal approximation using squashable activation functionscited by 3
  10. 2023Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networkscited by 2
  11. 2024Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machinescited by 2
  12. 2024Optimal Neural Network Approximation for High-Dimensional Continuous Functionscited by 2
14 more
  1. 2024Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networkscited by 2
  2. 2025Approximation properties of neural ODEscited by 2
  3. 2025Minimum Width of Deep Narrow Networks for Universal Approximationcited by 2
  4. 2026A holomorphic neural network framework for 3D boundary value problems governed by harmonic potentialscited by 2
  5. 2024Neural Networks Trained by Weight Permutation are Universal Approximatorscited by 1
  6. 2025Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regimecited by 1
  7. 2026Low-dimensional topology of deep neural networkscited by 1
  8. 20232-Cats: 2D Copula Approximating Transformscited by 0
  9. 2024Achieving interpretable machine learning by functional decomposition of black-box models into explainable predictor effectscited by 0
  10. 2025Vocabulary In-Context Learning in Transformers: Benefits of Positional Encodingcited by 0
  11. 2026Approximation Theory for Neural Networks: Old and Newcited by 0
  12. 2026Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximationscited by 0
  13. 2026Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width Onecited by 0
  14. 2026The minimal width of neural networks and immersions of real projective spacecited by 0