# Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm

arXiv: https://arxiv.org/abs/2207.12242 · HTML: https://mlsanity.com/paper/2207.12242/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 122 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 28 of 28)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [A survey of the Schr"odinger problem and some of its connections with optimal transport](https://mlsanity.com/paper/1308.0215/) (2013)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Improving GANs Using Optimal Transport](https://mlsanity.com/paper/1803.05573/) (2018)
- [On the relation between optimal transport and Schr"odinger bridges: A stochastic control viewpoint](https://mlsanity.com/paper/1412.4430/) (2014)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1902.00089/) (2019)
- [WaveFlow: A Compact Flow-based Model for Raw Audio](https://mlsanity.com/paper/1912.01219/) (2019)
- [From optimal transport to generative modeling: the VEGAN cookbook](https://mlsanity.com/paper/1705.07642/) (2017)
- [Entropic and displacement interpolation: a computational approach using the Hilbert metric](https://mlsanity.com/paper/1506.04255/) (2015)
- [Deep learning as optimal control problems: models and numerical methods](https://mlsanity.com/paper/1904.05657/) (2019)
- [Monge-Amp`ere Flow for Generative Modeling](https://mlsanity.com/paper/1809.10188/) (2018)
- [Discriminator optimal transport](https://mlsanity.com/paper/1910.06832/) (2019)
- [Controlling an Autonomous Vehicle with Deep Reinforcement Learning](https://mlsanity.com/paper/1909.12153/) (2019)
- [Generalization Properties of Optimal Transport GANs with Latent Distribution Learning](https://mlsanity.com/paper/2007.14641/) (2020)
- [Deep Diffeomorphic Normalizing Flows](https://mlsanity.com/paper/1810.03256/) (2018)
- [Variational Autoencoders with Normalizing Flow Decoders](https://mlsanity.com/paper/2004.05617/) (2020)
- [Deep Learning: Hydrodynamics, and Lie-Poisson Hamilton-Jacobi Theory](https://mlsanity.com/paper/2105.09542/) (2021)

## Abstract (arXiv)

As it stands, a robust mathematical framework to analyse and study various topics in deep learning is yet to come to the fore. Nonetheless, viewing deep learning as a dynamical system allows the use of established theories to investigate the behaviour of deep neural networks. In order to study the stability of the training process, in this article, we formulate training of deep neural networks as a hydrodynamics system, which has a multisymplectic structure. For that, the deep neural network is modelled using a stochastic differential equation and, thereby, mean-field type control is used to train it. The necessary conditions for optimality of the mean--field type control reduce to a system of Euler-Poincare equations, which has the a similar geometric structure to that of compressible fluids. The mean-field type control is solved numerically using a multisymplectic numerical scheme that takes advantage of the underlying geometry. Moreover, the numerical scheme, yields an approximated solution which is also an exact solution of a hydrodynamics system with a multisymplectic structure and it can be analysed using backward error analysis. Further, nonlinear stability yields the condition for selecting the number of hidden layers and the number of nodes per layer, that makes the training stable while approximating the solution of a residual neural network with a number of hidden layers approaching infinity.