mlsanityData to 7 Oct 2026

Representation Benefits of Deep Feedforward Networks

arXiv 1509.08101 ↗First posted 27 Sep 2015cs.LG cs.NEcited by 84

Not on the map. Why?

Abstract (arXiv)

This note provides a family of classification problems, indexed by a positive integer k, where all shallow networks with fewer than exponentially (in k) many nodes exhibit error at least 1/6, whereas a deep network with 2 nodes in each of 2k layers achieves zero error, as does a recurrent network with 3 distinct nodes iterated k times. The proof is elementary, and the networks are standard feedforward networks with ReLU (Rectified Linear Unit) nonlinearities.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 19; 2023: 21; 2024: 12; 2025: 9; 2026: 10.

84 papers.

Sort:

By area:

  1. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  2. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  3. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  4. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  5. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  6. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  7. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  8. 2023Deep networks for system identification: a Surveycited by 35
  9. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  10. 2022Selective Network Linearization for Efficient Private Inferencecited by 31
  11. 2021Multifidelity Modeling for Physics-Informed Neural Networks (PINNs)cited by 25
  12. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
38 more
  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  2. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  3. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  4. 2021On the Expected Complexity of Maxout Networkscited by 15
  5. 2021Neural Network Approximation of Refinable Functionscited by 15
  6. 2022Training Fully Connected Neural Networks is existsℝ-Completecited by 15
  7. 2024Simulating Hard Attention Using Soft Attentioncited by 15
  8. 2023Polyhedral Complex Extraction from ReLU Networks using Edge Subdivisioncited by 14
  9. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  10. 2022Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARLcited by 12
  11. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  12. 2021ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computationcited by 11
  13. 2024Optimization Over Trained Neural Networks: Taking a Relaxing Walkcited by 10
  14. 2022Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networkscited by 9
  15. 2022Curved Representation Space of Vision Transformerscited by 8
  16. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  17. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  18. 2023Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activationcited by 8
  19. 2021Universal Approximation of Residual Flows in Maximum Mean Discrepancycited by 7
  20. 2022Asymptotic Properties for Bayesian Neural Network in Besov Spacecited by 7
  21. 2023Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networkscited by 6
  22. 2024Variance reduction of diffusion model's gradients with Taylor approximation-based control variatecited by 6
  23. 2024When can classical neural networks represent quantum states?cited by 6
  24. 2023Deep ReLU Networks Have Surprisingly Simple Polytopescited by 5
  25. 2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4
  26. 2022Theoretical Exploration of Solutions of Feedforward ReLU Networkscited by 4
  27. 2022Approximation Power of Deep Neural Networks: an explanatory mathematical surveycited by 4
  28. 2023The R-mAtrIx Netcited by 4
  29. 2023Embeddings between Barron spaces with higher order activation functionscited by 4
  30. 2023Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Trainingcited by 4
  31. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 4
  32. 2022Training Thinner and Deeper Neural Networks: Jumpstart Regularizationcited by 3
  33. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
  34. 2022CNNs Avoid Curse of Dimensionality by Learning on Patchescited by 3
  35. 2022Blessing of Nonconvexity in Deep Linear Models: Depth Flattens the Optimization Landscape Around the True Solutioncited by 3
  36. 2022On Scrambling Phenomena for Randomly Initialized Recurrent Networkscited by 3
  37. 2023Understanding Parameter Sharing in Transformerscited by 3
  38. 2025On Space Folds of ReLU Neural Networkscited by 3

50 most cited of 84 All 84 (JSON)