# Variational formulations of ODE-Net as a mean-field optimal control problem and existence results

arXiv: https://arxiv.org/abs/2303.05924 · HTML: https://mlsanity.com/paper/2303.05924/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control](https://mlsanity.com/paper/1909.12077/) (2019)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth](https://mlsanity.com/paper/2003.05508/) (2020)
- [Momentum Residual Neural Networks](https://mlsanity.com/paper/2102.07870/) (2021)
- [Universal Approximation Property of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2012.02414/) (2020)
- [Do Residual Neural Networks discretize Neural Ordinary Differential Equations?](https://mlsanity.com/paper/2205.14612/) (2022)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [Mean-Field Neural ODEs via Relaxed Optimal Control](https://mlsanity.com/paper/1912.05475/) (2019)
- [On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting](https://mlsanity.com/paper/2106.06251/) (2021)
- [Large-time asymptotics in deep learning](https://mlsanity.com/paper/2008.02491/) (2020)
- [Stateful ODE-Nets using Basis Function Expansions](https://mlsanity.com/paper/2106.10820/) (2021)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [A Measure Theoretical Approach to the Mean-field Maximum Principle for Training NeurODEs](https://mlsanity.com/paper/2107.08707/) (2021)
- [Global convergence of ResNets: From finite to infinite width using linear parameterization](https://mlsanity.com/paper/2112.05531/) (2021)
- [A Shooting Formulation of Deep Learning](https://mlsanity.com/paper/2006.10330/) (2020)
- [Optimal Control of ensembles of dynamical systems](https://mlsanity.com/paper/2203.16485/) (2022)
- [Optimal Control of Continuity Equations](https://mlsanity.com/paper/1506.08932/) (2015)
- [Differential Inclusions in Wasserstein Spaces: The Cauchy-Lipschitz Framework](https://mlsanity.com/paper/2007.08906/) (2020)
- [Learning via nonlinear conjugate gradients and depth-varying neural ODEs](https://mlsanity.com/paper/2202.05766/) (2022)
- [Lagrangian discretization of variational mean field games](https://mlsanity.com/paper/2010.11519/) (2020)
- [On the Properties of the Value Function Associated to a Mean-Field Optimal Control Problem of Bolza Type](https://mlsanity.com/paper/2107.13912/) (2021)

## Built on by (most cited first, 3 of 3)

- [A convergence result of a continuous model of deep learning via a Lojasiewicz--Simon inequality](https://mlsanity.com/paper/2311.15365/) (2023)
- [Cluster-based classification with neural ODEs via control](https://mlsanity.com/paper/2312.13807/) (2023)
- [Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers](https://mlsanity.com/paper/2605.07772/) (2026)

## Abstract (arXiv)

This paper presents a mathematical analysis of ODE-Net, a continuum model of deep neural networks (DNNs). In recent years, Machine Learning researchers have introduced ideas of replacing the deep structure of DNNs with ODEs as a continuum limit. These studies regard the "learning" of ODE-Net as the minimization of a "loss" constrained by a parametric ODE. Although the existence of a minimizer for this minimization problem needs to be assumed, only a few studies have investigated its existence analytically in detail. In the present paper, the existence of a minimizer is discussed based on a formulation of ODE-Net as a measure-theoretic mean-field optimal control problem. The existence result is proved when a neural network, which describes a vector field of ODE-Net, is linear with respect to learnable parameters. The proof employs the measure-theoretic formulation combined with the direct method of Calculus of Variations. Secondly, an idealized minimization problem is proposed to remove the above linearity assumption. Such a problem is inspired by a kinetic regularization associated with the Benamou--Brenier formula and universal approximation theorems for neural networks. The proofs of these existence results use variational methods, differential equations, and mean-field optimal control theory. They will stand for a new analytic way to investigate the learning process of deep neural networks.