Improving Neural ODEs via Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
Neural Ordinary Differential Equations (Neural ODEs) construct the continuous dynamics of hidden units using ordinary differential equations specified by a neural network, demonstrating promising results on many tasks. However, Neural ODEs still do not perform well on image recognition tasks. The possible reason is that the one-hot encoding vector commonly used in Neural ODEs can not provide enough supervised information. We propose a new training based on knowledge distillation to construct more powerful and robust Neural ODEs fitting image recognition tasks. Specially, we model the training of Neural ODEs into a teacher-student learning process, in which we propose ResNets as the teacher model to provide richer supervised information. The experimental results show that the new training manner can improve the classification accuracy of Neural ODEs by 24% on CIFAR10 and 5% on SVHN. In addition, we also quantitatively discuss the effect of both knowledge distillation and time horizon in Neural ODEs on robustness against adversarial examples. The experimental analysis concludes that introducing the knowledge distillation and increasing the time horizon can improve the robustness of Neural ODEs against adversarial examples.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 10 Mar 2022.
Builds on
25 of 31 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2019Hamiltonian Neural Networkscited by 702
- 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
- 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
- 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
- 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
- 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 484
13 more
- 2019Augmented Neural ODEscited by 396
- 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
- 2019Neural Jump Stochastic Differential Equationscited by 147
- 2020OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transportcited by 143
- 2019On Robustness of Neural Ordinary Differential Equationscited by 83
- 2020Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flowscited by 60
- 2017Efficient Defenses Against Adversarial Attackscited by 53
- 2017Dense Associative Memory is Robust to Adversarial Inputscited by 45
- 2017Boosting Adversarial Attacks with Momentumcited by 23
- 2020Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradientscited by 22
- 2020Learning Dynamics Models with Stable Invariant Setscited by 18
- 2020MRI Image Reconstruction via Learning Optimization Using Neural ODEscited by 14
- 2020Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEscited by 4
Built on by
Citing papers per year: 2022: 1.
1 paper.