mlsanityData to 7 Oct 2026

Minimum Width for Universal Approximation

arXiv 2006.08859 ↗First posted 16 Jun 2020cs.LG stat.MLcited by 92

Not on the map. Why?

Abstract (arXiv)

The universal approximation property of width-bounded networks has been studied as a dual of classical universal approximation results on depth-bounded networks. However, the critical width enabling the universal approximation has not been exactly characterized in terms of the input dimension dx and the output dimension dy. In this work, we provide the first definitive result in this direction for networks using the ReLU activation functions: The minimum width required for the universal approximation of the Lp functions is exactly maxdx+1,dy. We also prove that the same conclusion does not hold for the uniform approximation with ReLU, but does hold with an additional threshold activation function. Our proof technique can be also used to derive a tighter upper bound on the minimum width required for the universal approximation using networks with general activation functions.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 18; 2023: 17; 2024: 24; 2025: 15; 2026: 11.

92 papers.

Sort:

By area:

  1. 2021How Attentive are Graph Attention Networks?cited by 818
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  3. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  4. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  5. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  6. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  7. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  8. 2024HardNet: Hard-Constrained Neural Networks with Universal Approximation Guaranteescited by 29
  9. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 25
  10. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  11. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  12. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 17
38 more
  1. 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16
  2. 2024A Survey on Universal Approximation Theoremscited by 15
  3. 2023Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximationcited by 13
  4. 2021A likelihood approach to nonparametric estimation of a singular distribution using deep generative modelscited by 12
  5. 2022Small Transformers Compute Universal Metric Embeddingscited by 12
  6. 2022Minimal Width for Universal Property of Deep RNNcited by 12
  7. 2022Width is Less Important than Depth in ReLU Neural Networkscited by 10
  8. 2024Optimization Over Trained Neural Networks: Taking a Relaxing Walkcited by 10
  9. 2024PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videoscited by 10
  10. 2022Spectrally Adapted Physics-Informed Neural Networks for Solving Unbounded Domain Problemscited by 9
  11. 2024On the Nonlinearity of Layer Normalizationcited by 9
  12. 2024Addressing common misinterpretations of KART and UAT in neural network literaturecited by 9
  13. 2023LU-Net: Invertible Neural Networks Based on Matrix Factorizationcited by 8
  14. 2023Improved weight initialization for deep and narrow feedforward neural networkcited by 7
  15. 2025Solving Empirical Bayes via Transformerscited by 7
  16. 2022Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networkscited by 6
  17. 2022Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?cited by 6
  18. 2023Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositionscited by 6
  19. 2023Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networkscited by 6
  20. 2024On the (In)Compatibility between Group Fairness and Individual Fairnesscited by 6
  21. 2024Expressive Power of ReLU and Step Networks under Floating-Point Operationscited by 5
  22. 2024Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysiscited by 5
  23. 2022Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systemscited by 4
  24. 2023The R-mAtrIx Netcited by 4
  25. 2023Approximation and interpolation of deep neural networkscited by 4
  26. 2024Beyond Derivative Pathology of PINNs: Variable Splitting Strategy with Convergence Analysiscited by 4
  27. 2023On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameterscited by 3
  28. 2023Universal approximation with complex-valued deep narrow neural networkscited by 3
  29. 2024Control of Medical Digital Twins with Artificial Neural Networkscited by 3
  30. 2024Minimum number of neurons in fully connected layers of a given neural network (the first approximation)cited by 3
  31. 2024On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmeticcited by 3
  32. 2023Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networkscited by 2
  33. 2024Defining Neural Network Architecture through Polytope Structures of Datasetcited by 2
  34. 2024Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machinescited by 2
  35. 2024Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networkscited by 2
  36. 2025Approximation properties of neural ODEscited by 2
  37. 2025Minimum width for universal approximation using squashable activation functionscited by 2
  38. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 2

50 most cited of 92 All 92 (JSON)