mlsanityData to 7 Oct 2026

Benefits of depth in neural networks

arXiv 1602.04485 ↗First posted 14 Feb 2016cs.LG cs.NE stat.MLcited by 248

Not on the map. Why?

Abstract (arXiv)

For any positive integer k, there exist neural networks with Theta(k3) layers, Theta(1) nodes per layer, and Theta(1) distinct parameters which can not be approximated by networks with O(k) layers unless they are exponentially large --- they must possess Omega(2k) nodes. This result is proved here for a class of nodes termed "semi-algebraic gates" which includes the common choices of ReLU, maximum, indicator, and piecewise polynomial functions, therefore establishing benefits of depth against not just standard networks with ReLU gates, but also convolutional networks with ReLU and maximization gates, sum-product networks, and boosted decision trees (in this last case with a stronger separation: Omega(2k3) total tree nodes are required).

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 53; 2022: 39; 2023: 33; 2024: 41; 2025: 40; 2026: 42.

248 papers.

Sort:

By area:

  1. 2022Transformers Learn Shortcuts to Automatacited by 260
  2. 2021Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networkscited by 219
  3. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  4. 2023Representational Strengths and Limitations of Transformerscited by 132
  5. 2021SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effectscited by 103
  6. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  7. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  8. 2024Transformers, parallel computation, and logarithmic depthcited by 75
  9. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  10. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  11. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  12. 2022Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spacescited by 50
38 more
  1. 2024Theoretical limitations of multi-layer Transformercited by 43
  2. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  3. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  4. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  5. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  6. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 35
  7. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  8. 2022ExSpliNet: An interpretable and expressive spline-based neural networkcited by 33
  9. 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
  10. 2021On the Existence of Universal Lottery Ticketscited by 30
  11. 2021On the Approximation Power of Two-Layer Networks of Random ReLUscited by 29
  12. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  13. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  14. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  15. 2021On the statistical complexity of quantum circuitscited by 25
  16. 2021Multifidelity Modeling for Physics-Informed Neural Networks (PINNs)cited by 25
  17. 2022How do noise tails impact on deep ReLU networks?cited by 25
  18. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 25
  19. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
  20. 2023VanillaNet: the Power of Minimalism in Deep Learningcited by 23
  21. 2024The Real Tropical Geometry of Neural Networkscited by 23
  22. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  23. 2024When Can Transformers Count to n?cited by 22
  24. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  25. 2022Exponentially Improving the Complexity of Simulating the Weisfeiler-Lehman Test with Graph Neural Networkscited by 20
  26. 2021The Connection Between Approximation, Depth Separation and Learnability in Neural Networkscited by 19
  27. 2021Depth separation beyond radial functionscited by 18
  28. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  29. 2025On the Expressiveness of Rational ReLU Neural Networks With Bounded Depthcited by 18
  30. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  31. 2021Optimization-Based Separations for Neural Networkscited by 17
  32. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 17
  33. 2021Partition of unity networks: deep hp-approximationcited by 15
  34. 2021On the Expected Complexity of Maxout Networkscited by 15
  35. 2021Neural Network Approximation of Refinable Functionscited by 15
  36. 2021Approximation Properties of Deep ReLU CNNscited by 15
  37. 2022Training Fully Connected Neural Networks is existsℝ-Completecited by 15
  38. 2021Layer Folding: Neural Network Depth Reduction using Activation Linearizationcited by 14

50 most cited of 248 All 248 (JSON)