mlsanityData to 7 Oct 2026

Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems

arXiv 2209.10909 ↗First posted 22 Sep 2022cs.LG cs.NA math.NAcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep learning has made significant applications in the field of data science and natural science. Some studies have linked deep neural networks to dynamic systems, but the network structure is restricted to the residual network. It is known that residual networks can be regarded as a numerical discretization of dynamic systems. In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of splitting method to solve differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

25 of 39 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  5. 2016Density estimation using Real NVPcited by 2,029
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  8. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  9. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  10. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  11. 2018Invertible Residual Networkscited by 364
  12. 2017Stable Architectures for Deep Neural Networkscited by 340
13 more
  1. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  2. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  3. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  4. 2019Universal Approximation with Deep Narrow Networkscited by 198
  5. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 120
  6. 2020Minimum Width for Universal Approximationcited by 92
  7. 2020Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximatorscited by 85
  8. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 78
  9. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  10. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  11. 2016PolyNet: A Pursuit of Structural Diversity in Very Deep Networkscited by 38
  12. 2018Deep, Skinny Neural Networks are not Universal Approximatorscited by 37
  13. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3.

4 papers.

  1. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 25
  2. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 17
  3. 2023Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximationcited by 13
  4. 2023Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositionscited by 6