mlsanityData to 7 Oct 2026

Deep vs. shallow networks : An approximation theory perspective

arXiv 1608.03287 ↗First posted 10 Aug 2016cs.LG math.FAcited by 63

Not on the map. Why?

Abstract (arXiv)

The paper briefy reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Neural Networks perform much better in function approximation problems than shallow, one-hidden layer architectures. The paper announces new results for a non-smooth activation function - the ReLU function - used in present-day neural networks, as well as for the Gaussian networks. We propose a new definition of relative dimension to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Aug 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 11; 2023: 15; 2024: 13; 2025: 9; 2026: 5.

63 papers.

Sort:

By area:

  1. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  2. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  3. 2023VanillaNet: the Power of Minimalism in Deep Learningcited by 23
  4. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  5. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  6. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  7. 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
  8. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14
  9. 2021On the Sample Complexity of Learning under Invariance and Geometric Stabilitycited by 13
  10. 2022To update or not to update? Neurons at equilibrium in deep modelscited by 13
  11. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  12. 2022Effects of Data Geometry in Early Deep Learningcited by 12
38 more
  1. 2023Rates of Approximation by ReLU Shallow Neural Networkscited by 12
  2. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  3. 2022Local approximation of operatorscited by 8
  4. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  5. 2022Understanding the Evolution of Linear Regions in Deep Reinforcement Learningcited by 7
  6. 2023Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometriescited by 6
  7. 2021Estimation of a regression function on a manifold by fully connected deep neural networkscited by 5
  8. 2023Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performancecited by 5
  9. 2021Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Netscited by 4
  10. 2022De Rham compatible Deep Neural Network FEMcited by 4
  11. 2022Approximation Power of Deep Neural Networks: an explanatory mathematical surveycited by 4
  12. 2023Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learningcited by 4
  13. 2023Tractability of approximation by general shallow networkscited by 4
  14. 2024Hypothesis Spaces for Deep Learningcited by 4
  15. 2024Deep Learning as Ricci Flowcited by 4
  16. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  17. 2023Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selectioncited by 3
  18. 2023Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signalscited by 3
  19. 2024Spectral complexity of deep neural networkscited by 3
  20. 2024Just How Flexible are Neural Networks in Practice?cited by 3
  21. 2025Comparison of generalised additive models and neural networks in applications: A systematic reviewcited by 3
  22. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networkscited by 2
  23. 2022Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activationcited by 2
  24. 2025Nonlocal techniques for the analysis of deep ReLU neural network approximationscited by 2
  25. 2025Information-Theoretic Framework for Understanding Modern Machine-Learningcited by 2
  26. 2025Superposed parameterised quantum circuitscited by 2
  27. 2021The Effects of Image Distribution and Task on Adversarial Robustnesscited by 1
  28. 2022Classification of Natural Language Processing Techniques for Requirements Engineeringcited by 1
  29. 2022The BUTTER Zone: An Empirical Study of Training Dynamics in Fully Connected Neural Networkscited by 1
  30. 2022Xtreme Margin: A Tunable Loss Function for Binary Classification Problemscited by 1
  31. 2023Approximation by non-symmetric networks for cross-domain learningcited by 1
  32. 2024Systematic construction of continuous-time neural networks for linear dynamical systemscited by 1
  33. 2024Understanding Encoder-Decoder Structures in Machine Learning Using Information Measurescited by 1
  34. 2024Federated Learning with Flexible Architecturescited by 1
  35. 2024Golden Ratio-Based Sufficient Dimension Reductioncited by 1
  36. 2025Exploring the Effect of DNN Depth on Adversarial Attacks in Network Intrusion Detection Systemscited by 1
  37. 2025Autonomous-Flow-Based Generationcited by 1
  38. 2021Structural Sievescited by 0

50 most cited of 63 All 63 (JSON)