mlsanityData to 7 Oct 2026

On the Expressive Power of Deep Neural Networks

arXiv 1606.05336 ↗First posted 16 Jun 2016stat.ML cs.AI cs.LGcited by 302

Not on the map. Why?

Abstract (arXiv)

We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings can be summarized as follows: (1) The complexity of the computed function grows exponentially with depth. (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights. (3) Regularizing on trajectory length (trajectory regularization) is a simpler alternative to batch normalization, with the same performance.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 58; 2023: 52; 2024: 51; 2025: 58; 2026: 33.

302 papers.

Sort:

By area:

  1. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  2. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  3. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  4. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  5. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  6. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  7. 2021Model Complexity of Deep Learning: A Surveycited by 64
  8. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  9. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  10. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  11. 2022Neural Architecture Search for Spiking Neural Networkscited by 52
  12. 2024Deep Networks Always Grok and Here is Whycited by 49
38 more
  1. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  2. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  3. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  4. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  5. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  6. 2022On the Power of Foundation Modelscited by 37
  7. 2023Lower Bounds on the Depth of Integral ReLU Neural Networks via Lattice Polytopescited by 36
  8. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  9. 2023Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theoremcited by 33
  10. 2021Neural Response Interpretation through the Lens of Critical Pathwayscited by 32
  11. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  12. 2024Counting Ability of Large Language Models and Impact of Tokenizationcited by 29
  13. 2025Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMscited by 29
  14. 2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
  15. 2022LieGG: Studying Learned Lie Group Generatorscited by 28
  16. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  17. 2021Multifidelity Modeling for Physics-Informed Neural Networks (PINNs)cited by 25
  18. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  19. 2021Causal Navigation by Continuous-time Neural Networkscited by 23
  20. 2024The Real Tropical Geometry of Neural Networkscited by 23
  21. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  22. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  23. 2023Effective Theory of Transformers at Initializationcited by 21
  24. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  25. 2021On the Robustness of Average Losses for Partial-Label Learningcited by 20
  26. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  27. 2022Exponentially Improving the Complexity of Simulating the Weisfeiler-Lehman Test with Graph Neural Networkscited by 20
  28. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19
  29. 2022UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographycited by 19
  30. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  31. 2021Depth separation beyond radial functionscited by 18
  32. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  33. 2023A neural operator-based surrogate solver for free-form electromagnetic inverse designcited by 18
  34. 2025On the Expressiveness of Rational ReLU Neural Networks With Bounded Depthcited by 18
  35. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  36. 2022Decomposed Linear Dynamical Systems (dLDS) for learning the latent components of neural dynamicscited by 16
  37. 2021On the Expected Complexity of Maxout Networkscited by 15
  38. 2021Neural Network Approximation of Refinable Functionscited by 15

50 most cited of 302 All 302 (JSON)