mlsanityData to 7 Oct 2026

Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

arXiv 1708.02691 ↗First posted 9 Aug 2017stat.ML cs.CG cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width wmin(d) so that ReLU nets of width wmin(d) (and arbitrary depth) can approximate any continuous function on the unit cube [0,1]d aribitrarily well? For ReLU nets near this minimal width, what can one say about the depth necessary to approximate a given function? Our approach to this paper is based on the observation that, due to the convexity of the ReLU activation, ReLU nets are particularly well-suited for representing convex functions. In particular, we prove that ReLU nets with width d+1 can approximate any continuous convex function of d variables arbitrarily well. These results then give quantitative depth estimates for the rate of approximation of any continuous scalar function on the d-dimensional cube [0,1]d by ReLU nets with width d+3.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 21; 2023: 18; 2024: 16; 2025: 18; 2026: 6.

104 papers.

Sort:

By area:

  1. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  2. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  3. 2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
  4. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  5. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  6. 2023AbDiffuser: Full-Atom Generation of in vitro Functioning Antibodiescited by 35
  7. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  8. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 25
  9. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
  10. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  11. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  12. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 21
38 more
  1. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  2. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  3. 2022UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographycited by 19
  4. 2022ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Frameworkcited by 19
  5. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
  6. 2025On the Expressiveness of Rational ReLU Neural Networks With Bounded Depthcited by 18
  7. 2021Universal Consistency of Deep Convolutional Neural Networkscited by 16
  8. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
  9. 2022Training Fully Connected Neural Networks is existsℝ-Completecited by 15
  10. 2021On minimal representations of shallow ReLU networkscited by 13
  11. 2022MultiAuto-DeepONet: A Multi-resolution Autoencoder DeepONet for Nonlinear Dimension Reduction, Uncertainty Quantification and Operator Learning of Forward and Inverse Stochastic Problemscited by 13
  12. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  13. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  14. 2024Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisationcited by 12
  15. 2021ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computationcited by 11
  16. 2021A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundationscited by 11
  17. 2021Multi-Head ReLU Implicit Neural Representation Networkscited by 10
  18. 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 10
  19. 2024PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videoscited by 10
  20. 2021Bangla Natural Language Processing: A Comprehensive Analysis of Classical, Machine Learning, and Deep Learning Based Methodscited by 8
  21. 2021Characterizing Learning Dynamics of Deep Neural Networks via Complex Networkscited by 8
  22. 2021Universal Approximation of Residual Flows in Maximum Mean Discrepancycited by 7
  23. 2025FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimationcited by 7
  24. 2021Ensuring DNN Solution Feasibility for Optimization Problems with Convex Constraints and Its Application to DC Optimal Power Flow Problemscited by 6
  25. 2022Reachable Polyhedral Marching (RPM): An Exact Analysis Tool for Deep-Learned Control Systemscited by 6
  26. 2022On Solution Functions of Optimization: Universal Approximation and Covering Number Boundscited by 6
  27. 2022Bayesian Bilinear Neural Network for Predicting the Mid-price Dynamics in Limit-Order Book Marketscited by 5
  28. 2022A Manifold Two-Sample Test Study: Integral Probability Metric with Neural Networkscited by 5
  29. 2023Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performancecited by 5
  30. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  31. 2021Finding Everything within Random Binary Networkscited by 4
  32. 2023Formalizing Piecewise Affine Activation Functions of Neural Networks in Coqcited by 4
  33. 2023Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Trainingcited by 4
  34. 2024Lower Bounds on the Expressivity of Recurrent Neural Language Modelscited by 4
  35. 2025Orthogonal Representation Learning for Estimating Causal Quantitiescited by 4
  36. 2025Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Timecited by 4
  37. 2024Learning solution operators of PDEs defined on varying domains via MIONetcited by 3
  38. 2024Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimationcited by 3

50 most cited of 104 All 104 (JSON)