mlsanityData to 7 Oct 2026

The power of deeper networks for expressing natural functions

arXiv 1705.05502 ↗First posted 16 May 2017cs.LG cs.NE stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons m required to approximate natural classes of multivariate polynomials of n variables grows only linearly with n for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from 1 to k, the neuron requirement grows exponentially not with n but with n1/k, suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with n.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 5; 2023: 8; 2024: 2; 2025: 2; 2026: 4.

35 papers.

Sort:

  1. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  2. 2021On the approximation of functions by tanh neural networkscited by 81
  3. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  4. 2021Fast convolutional neural networks on FPGAs with hls4mlcited by 42
  5. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  6. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  7. 2021Depth separation beyond radial functionscited by 18
  8. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  9. 2021Explicitly antisymmetrized neural network layers for variational Monte Carlo simulationcited by 14
  10. 2021Reverse Engineering the Neural Tangent Kernelcited by 12
  11. 2022Minimal Width for Universal Property of Deep RNNcited by 12
  12. 2021Connections between Numerical Algorithms for PDEs and Neural Networkscited by 10
23 more
  1. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8
  2. 2023More than Correlation: Do Large Language Models Learn Causal Representations of Space?cited by 7
  3. 2022Neural Networks as Paths through the Space of Representationscited by 5
  4. 2022Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximationcited by 4
  5. 2023The Evolution of the Interplay Between Input Distributions and Linear Regions in Networkscited by 3
  6. 2023Artificial Neural Nets and the Representation of Human Conceptscited by 3
  7. 2021Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learningcited by 2
  8. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networkscited by 2
  9. 2024Depth Separation in Norm-Bounded Infinite-Width Neural Networkscited by 2
  10. 2021Strategic bidding in freight transport using deep reinforcement learningcited by 1
  11. 2022Inference on Time Series Nonparametric Conditional Moment Restrictions Using General Sievescited by 1
  12. 2023Expressivity of Shallow and Deep Neural Networks for Polynomial Approximationcited by 1
  13. 2025System Identification and Control Using Lyapunov-Based Deep Neural Networks without Persistent Excitation: A Concurrent Learning Approachcited by 1
  14. 2026Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networkscited by 1
  15. 2021MAGI-X: Manifold-Constrained Gaussian Process Inference for Unknown System Dynamicscited by 0
  16. 2023MOSAIC, acomparison framework for machine learning modelscited by 0
  17. 2023Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Losscited by 0
  18. 2023Reverse Engineering Deep ReLU Networks An Optimization-based Algorithmcited by 0
  19. 2024Structured Deep Neural Network-Based Backstepping Trajectory Tracking Control for Lagrangian Systemscited by 0
  20. 2025MIDUS: Memory-Infused Depth Up-Scalingcited by 0
  21. 2026Efficient Aspect Term Extraction using Spiking Neural Networkcited by 0
  22. 2026On the Role of Depth in the Expressivity of RNNscited by 0
  23. 2026Complexity of Linear Regions in Self-supervised Deep ReLU Networkscited by 0