mlsanityData to 7 Oct 2026

Depth Separation with Multilayer Mean-Field Networks

arXiv 2304.01063 ↗First posted 3 Apr 2023cs.LG math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

Depth separation -- why a deeper network is more powerful than a shallower one -- has been a major problem in deep learning theory. Previous results often focus on representation power. For example, arXiv:1904.06984 constructed a function that is easy to approximate using a 3-layer network but not approximable by any 2-layer network. In this paper, we show that this separation is in fact algorithmic: one can learn the function constructed by arXiv:1904.06984 using an overparameterized network with polynomially many neurons efficiently. Our result relies on a new way of extending the mean-field limit to multilayer networks, and a decomposition of loss that factors out the error introduced by the discretization of infinite-width mean-field networks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

44 of 52 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2016Error bounds for approximations with deep ReLU networkscited by 567
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  10. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  11. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  12. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
32 more
  1. 2016Benefits of depth in neural networkscited by 248
  2. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  3. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  4. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  5. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  6. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  7. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  8. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  9. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  10. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  11. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  12. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  13. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  14. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  15. 2016Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networkscited by 58
  16. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  17. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  18. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  19. 2017Depth Separation for Neural Networkscited by 46
  20. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  21. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  22. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  23. 2019Depth Separations in Neural Networks: What is Actually Being Separated?cited by 27
  24. 2019Is Deeper Better only when Shallow is Good?cited by 24
  25. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  26. 2021The Connection Between Approximation, Depth Separation and Learnability in Neural Networkscited by 19
  27. 2021Depth separation beyond radial functionscited by 18
  28. 2020Neural Networks with Small Weights and Depth-Separation Barrierscited by 17
  29. 2021Optimization-Based Separations for Neural Networkscited by 17
  30. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  31. 2020Beyond Lazy Training for Over-parameterized Tensor Decompositioncited by 15
  32. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 0; 2026: 2.

8 papers.

  1. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  2. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  3. 2023Learning Hierarchical Polynomials with Three-Layer Neural Networkscited by 8
  4. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  5. 2026Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chainingcited by 5
  6. 2024Depth Separations in Neural Networks: Separating the Dimension from the Accuracycited by 4
  7. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  8. 2026Algorithmic Separation between Constant-Depth and Logarithmic-Depth Neural Networkscited by 0