# Simple Video Generation using Neural ODEs

arXiv: https://arxiv.org/abs/2109.03292 · HTML: https://mlsanity.com/paper/2109.03292/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Adversarial Video Generation on Complex Datasets](https://mlsanity.com/paper/1907.06571/) (2019)

## Built on by (most cited first, 9 of 9)

- [Modulated Neural ODEs](https://mlsanity.com/paper/2302.13262/) (2023)
- [Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE](https://mlsanity.com/paper/2303.05323/) (2023)
- [Longitudinal Flow Matching for Trajectory Modeling](https://mlsanity.com/paper/2510.03569/) (2025)
- [SMURF: Continuous Dynamics for Motion-Deblurring Radiance Fields](https://mlsanity.com/paper/2403.07547/) (2024)
- [VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs](https://mlsanity.com/paper/2304.06020/) (2023)
- [Modelling Latent Dynamics of StyleGAN using Neural ODEs](https://mlsanity.com/paper/2208.11197/) (2022)
- [Variational Inference for Longitudinal Data Using Normalizing Flows](https://mlsanity.com/paper/2303.14220/) (2023)
- [Exploiting Inductive Biases in Video Modeling through Neural CDEs](https://mlsanity.com/paper/2311.04986/) (2023)
- [NeuralPrefix: A Zero-shot Sensory Data Imputation Plugin](https://mlsanity.com/paper/2502.05883/) (2025)

## Abstract (arXiv)

Despite having been studied to a great extent, the task of conditional generation of sequences of frames, or videos, remains extremely challenging. It is a common belief that a key step towards solving this task resides in modelling accurately both spatial and temporal information in video signals. A promising direction to do so has been to learn latent variable models that predict the future in latent space and project back to pixels, as suggested in recent literature. Following this line of work and building on top of a family of models introduced in prior work, Neural ODE, we investigate an approach that models time-continuous dynamics over a continuous latent space with a differential equation with respect to time. The intuition behind this approach is that these trajectories in latent space could then be extrapolated to generate video frames beyond the time steps for which the model is trained. We show that our approach yields promising results in the task of future frame prediction on the Moving MNIST dataset with 1 and 2 digits.