# From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks

arXiv: https://arxiv.org/abs/2307.02279 · HTML: https://mlsanity.com/paper/2307.02279/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Convolutional Neural Networks at Constrained Time Cost](https://mlsanity.com/paper/1412.1710/) (2014)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Deep learning as optimal control problems: models and numerical methods](https://mlsanity.com/paper/1904.05657/) (2019)
- [More is Less: Inducing Sparsity via Overparameterization](https://mlsanity.com/paper/2112.11027/) (2021)
- [Learning Manifold Dimensions with Conditional Variational Autoencoders](https://mlsanity.com/paper/2302.11756/) (2023)
- [Large-time asymptotics in deep learning](https://mlsanity.com/paper/2008.02491/) (2020)
- [A Measure Theoretical Approach to the Mean-field Maximum Principle for Training NeurODEs](https://mlsanity.com/paper/2107.08707/) (2021)
- [Turnpike in optimal control of PDEs, ResNets, and beyond](https://mlsanity.com/paper/2202.04097/) (2022)
- [Designing Stable Neural Networks using Convex Analysis and ODEs](https://mlsanity.com/paper/2306.17332/) (2023)
- [Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems](https://mlsanity.com/paper/2110.12393/) (2021)
- [Selection dynamics for deep neural networks](https://mlsanity.com/paper/1905.09076/) (2019)
- [A gradient flow equation for optimal control problems with end-point cost](https://mlsanity.com/paper/2107.00556/) (2021)
- [Neural Architecture Search via Bregman Iterations](https://mlsanity.com/paper/2106.02479/) (2021)
- [Optimal Control of ensembles of dynamical systems](https://mlsanity.com/paper/2203.16485/) (2022)
- [Intrinsic dimension estimation for discrete metrics](https://mlsanity.com/paper/2207.09688/) (2022)
- [Viability and Exponentially Stable Trajectories for Differential Inclusions in Wasserstein Spaces](https://mlsanity.com/paper/2209.03640/) (2022)
- [On the Viability and Invariance of Proper Sets under Continuity Inclusions in Wasserstein Spaces](https://mlsanity.com/paper/2304.11945/) (2023)

## Built on by (most cited first, 3 of 3)

- [A minimax optimal control approach for robust neural ODEs](https://mlsanity.com/paper/2310.17584/) (2023)
- [Normalizing flows as approximations of optimal transport maps via linear-control neural ODEs](https://mlsanity.com/paper/2311.01404/) (2023)
- [Constructive interpolation and generalization rates for neural ODEs: a control perspective](https://mlsanity.com/paper/2606.00469/) (2026)

## Abstract (arXiv)

The connection between Residual Neural Networks (ResNets) and continuous-time control systems (known as NeurODEs) has led to a mathematical analysis of neural networks which has provided interesting results of both theoretical and practical significance. However, by construction, NeurODEs have been limited to describing constant-width layers, making them unsuitable for modeling deep learning architectures with layers of variable width. In this paper, we propose a continuous-time Autoencoder, which we call AutoencODE, based on a modification of the controlled field that drives the dynamics. This adaptation enables the extension of the mean-field control framework originally devised for conventional NeurODEs. In this setting, we tackle the case of low Tikhonov regularization, resulting in potentially non-convex cost landscapes. While the global results obtained for high Tikhonov regularization may not hold globally, we show that many of them can be recovered in regions where the loss function is locally convex. Inspired by our theoretical findings, we develop a training method tailored to this specific type of Autoencoders with residual connections, and we validate our approach through numerical experiments conducted on various examples.