mlsanityData to 7 Oct 2026

Deep limits and cut-off phenomena for neural networks

arXiv 2104.10727 ↗First posted 21 Apr 2021cs.LG math.DS stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider dynamical and geometrical aspects of deep learning. For many standard choices of layer maps we display semi-invariant metrics which quantify differences between data or decision functions. This allows us, when considering random layer maps and using non-commutative ergodic theorems, to deduce that certain limits exist when letting the number of layers tend to infinity. We also examine the random initialization of standard networks where we observe a surprising cut-off phenomenon in terms of the number of layers, the depth of the network. This could be a relevant parameter when choosing an appropriate number of layers for a given learning task, or for selecting a good initialization procedure. More generally, we hope that the notions and results in this paper can provide a framework, in particular a geometric one, for a part of the theoretical understanding of deep neural networks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 45 references that are in the map. First posted on arXiv 21 Apr 2021.

Builds on

16 of 45 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  8. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  9. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 168
  10. 2019Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noisecited by 98
  11. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  12. 2019Machine Learning from a Continuous Viewpointcited by 50
4 more
  1. 2014Avoiding pathologies in very deep networkscited by 33
  2. 2020Wide Neural Networks with Bottlenecks are Deep Gaussian Processescited by 18
  3. 2019Neural ODEs as the Deep Limit of ResNets with constant weightscited by 10
  4. 2018Analysis on the Nonlinear Dynamics of Deep Neural Networks: Topological Entropy and Chaoscited by 1

Built on by

Not cited yet.