mlsanityData to 14 Oct 2026

The Power of Depth for Feedforward Neural Networks

arXiv 1512.03965 ↗First posted 12 Dec 2015cs.LG cs.NE stat.MLcited by 183

Not on the map. Why?

Abstract (arXiv)

We show that there is a simple (approximately radial) function on realsd, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all known activation functions, including rectified linear units, sigmoids and thresholds, and formally demonstrates that depth -- even if increased by 1 -- can be exponentially more valuable than width for standard feedforward neural networks. Moreover, compared to related results in the context of Boolean functions, our result requires fewer assumptions, and the proof techniques and construction are very different.

Key facts. Cited by 183 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 48; 2022: 37; 2023: 29; 2024: 28; 2025: 17; 2026: 24.

183 papers.

Sort:

By area:

  1. 2022Transformers Learn Shortcuts to Automatacited by 268
  2. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  3. 2023Representational Strengths and Limitations of Transformerscited by 133
  4. 2021SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effectscited by 104
  5. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  6. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  7. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  8. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 56
  9. 2024Theoretical limitations of multi-layer Transformercited by 45
  10. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  11. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 39
  12. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
38 more
  1. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  2. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  3. 2021On the Approximation Power of Two-Layer Networks of Random ReLUscited by 29
  4. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  5. 2023VanillaNet: the Power of Minimalism in Deep Learningcited by 23
  6. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  7. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  8. 2021The Connection Between Approximation, Depth Separation and Learnability in Neural Networkscited by 19
  9. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
  10. 2021Depth separation beyond radial functionscited by 18
  11. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 18
  12. 2025On the Expressiveness of Rational ReLU Neural Networks With Bounded Depthcited by 18
  13. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  14. 2021Optimization-Based Separations for Neural Networkscited by 17
  15. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  16. 2022Bootstrap Generalization Ability from Loss Landscape Perspectivecited by 16
  17. 2021Neural Network Approximation of Refinable Functionscited by 15
  18. 2021Data-driven approaches for predicting spread of infectious diseases through DINNs: Disease Informed Neural Networkscited by 15
  19. 2022Training Fully Connected Neural Networks is existsℝ-Completecited by 15
  20. 2025The Serial Scaling Hypothesiscited by 13
  21. 2021ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computationcited by 12
  22. 2021Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networkscited by 12
  23. 2021Neural networks with linear threshold activations: structure and algorithmscited by 12
  24. 2021A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentationcited by 12
  25. 2023How Many Neurons Does it Take to Approximate the Maximum?cited by 12
  26. 2024Locality Sensitive Sparse Encoding for Learning World Models Onlinecited by 12
  27. 2025Better Neural Network Expressivity: Subdividing the Simplexcited by 12
  28. 2024On Minimal Depth in Neural Networkscited by 11
  29. 2025Depth-Bounds for Neural Networks via the Braid Arrangementcited by 11
  30. 2025On the Depth of Monotone ReLU Neural Networks and ICNNscited by 11
  31. 2022Width is Less Important than Depth in ReLU Neural Networkscited by 10
  32. 2022Exponential Separations in Symmetric Neural Networkscited by 10
  33. 2022Size and depth of monotone neural networks: interpolation and approximationcited by 10
  34. 2022Variational Inference for Infinitely Deep Neural Networkscited by 10
  35. 2022Non-Linear Coordination Graphscited by 10
  36. 2023Topological Expressivity of ReLU Neural Networkscited by 10
  37. 2023Depth Separation with Multilayer Mean-Field Networkscited by 9
  38. 2023Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activationcited by 9

50 most cited of 183 All 183 (JSON)