mlsanityData to 7 Oct 2026

Exploring the Complexity of Deep Neural Networks through Functional Equivalence

arXiv 2305.11417 ↗First posted 19 May 2023cs.LG math.ST stat.THcited by 4

Not on the map. Why?

Abstract (arXiv)

We investigate the complexity of deep neural networks through the lens of functional equivalence, which posits that different parameterizations can yield the same network function. Leveraging the equivalence property, we present a novel bound on the covering number for deep neural networks, which reveals that the complexity of neural networks can be reduced. Additionally, we demonstrate that functional equivalence benefits optimization, as overparameterized networks tend to be easier to train since increasing network width leads to a diminishing volume of the effective parameter space. These findings can offer valuable insights into the phenomenon of overparameterization and have implications for understanding generalization and optimization in deep learning.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 52 of its 79 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

52 of 79 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2016Error bounds for approximations with deep ReLU networkscited by 567
38 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  3. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  5. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  6. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  7. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  12. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  13. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  14. 2020Deep Network Approximation for Smooth Functionscited by 163
  15. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  16. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  17. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  18. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  19. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  20. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  21. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  22. 2017Implicit Regularization in Deep Learningcited by 90
  23. 2019Reverse-Engineering Deep ReLU Networkscited by 81
  24. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  25. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  26. 2017Riemannian approach to batch normalizationcited by 35
  27. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 26
  28. 2018On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyondcited by 24
  29. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  30. 2020Theory of Deep Convolutional Neural Networks II: Spherical Analysiscited by 22
  31. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  32. 2019Generalization Bounds for Convolutional Neural Networkscited by 21
  33. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  34. 2022Functional dimension of feedforward ReLU neural networkscited by 20
  35. 2015Understanding symmetries in deep networkscited by 17
  36. 2019Equi-normalization of Neural Networkscited by 16
  37. 2015Symmetry-invariant optimization in deep networkscited by 15
  38. 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Belted and Ensembled Neural Network for Linear and Nonlinear Sufficient Dimension Reductioncited by 4
  2. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  3. 2025A Theory for Conditional Generative Modeling on Multiple Data Sourcescited by 1
  4. 2026Behavioral Capacity Certificates for Quantized Language Modelscited by 0