# Imbedding Deep Neural Networks

arXiv: https://arxiv.org/abs/2202.00113 · HTML: https://mlsanity.com/paper/2202.00113/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [ODE2VAE: Deep generative second order ODEs with Bayesian neural networks](https://mlsanity.com/paper/1905.10994/) (2019)
- [A Shooting Formulation of Deep Learning](https://mlsanity.com/paper/2006.10330/) (2020)
- [Time Dependence in Non-Autonomous Neural ODEs](https://mlsanity.com/paper/2005.01906/) (2020)
- [Machine Learning and Control Theory](https://mlsanity.com/paper/2006.05604/) (2020)

## Built on by (most cited first, 1 of 1)

- [Rademacher Complexity of Neural ODEs via Chen-Fliess Series](https://mlsanity.com/paper/2401.16655/) (2024)

## Abstract (arXiv)

Continuous-depth neural networks, such as Neural ODEs, have refashioned the understanding of residual neural networks in terms of non-linear vector-valued optimal control problems. The common solution is to use the adjoint sensitivity method to replicate a forward-backward pass optimisation problem. We propose a new approach which explicates the network's `depth' as a fundamental variable, thus reducing the problem to a system of forward-facing initial value problems. This new method is based on the principle of `Invariant Imbedding' for which we prove a general solution, applicable to all non-linear, vector-valued optimal control problems with both running and terminal loss. Our new architectures provide a tangible tool for inspecting the theoretical--and to a great extent unexplained--properties of network depth. They also constitute a resource of discrete implementations of Neural ODEs comparable to classes of imbedded residual neural networks. Through a series of experiments, we show the competitive performance of the proposed architectures for supervised learning and time series prediction.