mlsanityData to 7 Oct 2026

Neural Generalized Ordinary Differential Equations with Layer-varying Parameters

arXiv 2209.10633 ↗First posted 21 Sep 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Deep residual networks (ResNets) have shown state-of-the-art performance in various real-world applications. Recently, the ResNets model was reparameterized and interpreted as solutions to a continuous ordinary differential equation or Neural-ODE model. In this study, we propose a neural generalized ordinary differential equation (Neural-GODE) model with layer-varying parameters to further extend the Neural-ODE to approximate the discrete ResNets. Specifically, we use nonparametric B-spline functions to parameterize the Neural-GODE so that the trade-off between the model complexity and computational efficiency can be easily balanced. It is demonstrated that ResNets and Neural-ODE models are special cases of the proposed Neural-GODE model. Based on two benchmark datasets, MNIST and CIFAR-10, we show that the layer-varying Neural-GODE is more flexible and general than the standard Neural-ODE. Furthermore, the Neural-GODE enjoys the computational and memory benefits while performing comparably to ResNets in prediction accuracy.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 62 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

26 of 62 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2018Group Normalizationcited by 1,514
  9. 2019Hamiltonian Neural Networkscited by 702
  10. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  11. 2019Deep Equilibrium Modelscited by 603
  12. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
14 more
  1. 2019Augmented Neural ODEscited by 396
  2. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  3. 2017Stable Architectures for Deep Neural Networkscited by 340
  4. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  5. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  6. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  7. 2020Dissecting Neural ODEscited by 150
  8. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  9. 2017Maximum Principle Based Algorithms for Deep Learningcited by 77
  10. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  11. 2021Noisy Recurrent Neural Networkscited by 42
  12. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  13. 2016Residual Networks of Residual Networks: Multilevel Residual Networkscited by 13
  14. 2020Understanding Recurrent Neural Networks Using Nonequilibrium Response Theorycited by 10

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Generalization bounds for neural ordinary differential equations and deep residual networkscited by 21