Meta-Solver for Neural Ordinary Differential Equations
Not on the map. Why?
Abstract (arXiv)
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization method and its properties, which may not be optimal for the selected application and yield the overfitting to the given solver. In our paper, we investigate how the variability in solvers' space can improve neural ODEs performance. We consider a family of Runge-Kutta methods that are parameterized by no more than two scalar variables. Based on the solvers' properties, we propose an approach to decrease neural ODEs overfitting to the pre-defined solver, along with a criterion to evaluate such behaviour. Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks. Recently it was shown that neural ODEs demonstrate superiority over conventional CNNs in terms of robustness. Our work demonstrates that the model robustness can be further improved by optimizing solver choice for a given task. The source code to reproduce our experiments is available at https://github.com/juliagusak/neural-ode-metasolver.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 15 Mar 2021.
Builds on
31 of 48 references.
References · 31
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2020Meta-Learning in Neural Networks: A Surveycited by 944
- 2020Fast is better than free: Revisiting adversarial trainingcited by 723
- 2019Hamiltonian Neural Networkscited by 702
- 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
- 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
- 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
- 2019Augmented Neural ODEscited by 396
- 2020Scalable Gradients for Stochastic Differential Equationscited by 264
- 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
- 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
- 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
19 more
- 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
- 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
- 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 168
- 2019Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noisecited by 98
- 2018Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systemscited by 88
- 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 83
- 2019On Robustness of Neural Ordinary Differential Equationscited by 83
- 2020Learning Differential Equations that are Easy to Solvecited by 81
- 2020Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODEcited by 76
- 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
- 2020STEER: Simple Temporal Regularization For Neural ODEscited by 58
- 2020Continuous-in-Depth Neural Networkscited by 40
- 2016PolyNet: A Pursuit of Structural Diversity in Very Deep Networkscited by 38
- 2020Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flowscited by 36
- 2020MRI Image Reconstruction via Learning Optimization Using Neural ODEscited by 14
- 2020Stochasticity in Neural ODEs: An Empirical Studycited by 13
- 2020Towards Understanding Normalization in Neural ODEscited by 7
- 2020Interpolation Technique to Speed Up Gradients Propagation in Neural ODEscited by 6
- 2020Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEscited by 4
Built on by
Citing papers per year: 2021: 1.
1 paper.