mlsanityData to 7 Oct 2026

Universal Approximation with Deep Narrow Networks

arXiv 1905.08539 ↗First posted 21 May 2019cs.LG math.CA stat.MLcited by 198

Not on the map. Why?

Abstract (arXiv)

The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonaffine continuous function, with a continuous nonzero derivative at some point. Then we show that the class of neural networks of arbitrary depth, width n + m + 2, and activation function ρ, is dense in C(K; ℝm) for K subseteq ℝn with K compact. This covers every activation function possible to use in practice, and also includes polynomial activation functions, which is unlike the classical version of the theorem, and provides a qualitative difference between deep narrow networks and shallow wide networks. We then consider several extensions of this result. In particular we consider nowhere differentiable activation functions, density in noncompact domains with respect to the Lp-norm, and how the width may be reduced to just n + m + 1 for `most' activation functions.

Key facts. Cited by 198 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 30; 2023: 29; 2024: 40; 2025: 38; 2026: 25.

198 papers.

Sort:

By area:

  1. 2021Neural SDEs as Infinite-Dimensional GANscited by 159
  2. 2022Neural Optimal Transportcited by 123
  3. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  4. 2021Efficient and Accurate Gradients for Neural SDEscited by 68
  5. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  6. 2024fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functionscited by 60
  7. 2021Variational Wasserstein gradient flowcited by 53
  8. 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 42
  9. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  10. 2021LEADS: Learning Dynamical Systems that Generalize Across Environmentscited by 39
  11. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  12. 2022Constrained Monotonic Neural Networkscited by 31
38 more
  1. 2024Applied Causal Inference Powered by ML and AIcited by 30
  2. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  3. 2024HardNet: Hard-Constrained Neural Networks with Universal Approximation Guaranteescited by 29
  4. 2022LieGG: Studying Learned Lie Group Generatorscited by 28
  5. 2021Machine-Learning Mathematical Structurescited by 25
  6. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 25
  7. 2024Evaluating the design space of diffusion-based generative modelscited by 25
  8. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  9. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  10. 2022Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysiscited by 22
  11. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  12. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  13. 2022UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographycited by 19
  14. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  15. 2021Physics-enhanced deep surrogates for partial differential equationscited by 18
  16. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  17. 2025Kolmogorov-Arnold Fourier Networkscited by 18
  18. 2025On the Expressiveness of Rational ReLU Neural Networks With Bounded Depthcited by 18
  19. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 17
  20. 2023Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirscited by 15
  21. 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
  22. 2021Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networkscited by 12
  23. 2022A general approximation lower bound in Lp norm, with applications to feed-forward neural networkscited by 12
  24. 2022Small Transformers Compute Universal Metric Embeddingscited by 12
  25. 2022Minimal Width for Universal Property of Deep RNNcited by 12
  26. 2023Finite-time Lyapunov exponents of deep neural networkscited by 12
  27. 2023Energy-Guided Continuous Entropic Barycenter Estimation for General Costscited by 12
  28. 2024Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisationcited by 12
  29. 2021A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundationscited by 11
  30. 2023Data pruning and neural scaling laws: fundamental limitations of score-based algorithmscited by 11
  31. 2023Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activationscited by 11
  32. 2024PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representationscited by 11
  33. 2022Width is Less Important than Depth in ReLU Neural Networkscited by 10
  34. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  35. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  36. 2024Polynomial Composition Activations: Unleashing the Dynamics of Large Language Modelscited by 9
  37. 2023LU-Net: Invertible Neural Networks Based on Matrix Factorizationcited by 8
  38. 2024Separable Physics-informed Neural Networks for Solving the BGK Model of the Boltzmann Equationcited by 8

50 most cited of 198 All 198 (JSON)