# Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videos

arXiv: https://arxiv.org/abs/2106.04776 · HTML: https://mlsanity.com/paper/2106.04776/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 74 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Sparse identification of nonlinear dynamics for model predictive control in the low-data limit](https://mlsanity.com/paper/1711.05501/) (2017)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [SINDy-PI: A Robust Algorithm for Parallel Implicit Sparse Identification of Nonlinear Dynamics](https://mlsanity.com/paper/2004.02322/) (2020)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning](https://mlsanity.com/paper/1710.05741/) (2017)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Sparse learning of stochastic dynamic equations](https://mlsanity.com/paper/1712.02432/) (2017)
- [Label-Free Supervision of Neural Networks with Physics and Domain Knowledge](https://mlsanity.com/paper/1609.05566/) (2016)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [Learning Visual Predictive Models of Physics for Playing Billiards](https://mlsanity.com/paper/1511.07404/) (2015)
- [Reasoning About Physical Interactions with Object-Oriented Prediction and Planning](https://mlsanity.com/paper/1812.10972/) (2018)
- [Extracting Sparse High-Dimensional Dynamics from Limited Data](https://mlsanity.com/paper/1707.08528/) (2017)
- [On the Convergence of the SINDy Algorithm](https://mlsanity.com/paper/1805.06445/) (2018)
- [A unified sparse optimization framework to learn parsimonious physics-informed models from data](https://mlsanity.com/paper/1906.10612/) (2019)
- [Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video](https://mlsanity.com/paper/1905.11169/) (2019)
- [Structured Object-Aware Physics Prediction for Video Modeling and Planning](https://mlsanity.com/paper/1910.02425/) (2019)
- [Automatic Differentiation to Simultaneously Identify Nonlinear Dynamics and Extract Noise Probability Distributions from Data](https://mlsanity.com/paper/2009.08810/) (2020)
- [A Unified Framework for Sparse Relaxed Regularized Regression: SR3](https://mlsanity.com/paper/1807.05411/) (2018)
- [Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images](https://mlsanity.com/paper/1511.04048/) (2015)
- [Symbolic Pregression: Discovering Physical Laws from Distorted Video](https://mlsanity.com/paper/2005.11212/) (2020)
- [Numerical differentiation of noisy data: A unifying multi-objective optimization framework](https://mlsanity.com/paper/2009.01911/) (2020)

## Built on by (most cited first, 1 of 1)

- [Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation](https://mlsanity.com/paper/2507.06830/) (2025)

## Abstract (arXiv)

Distilling analytical models from data has the potential to advance our understanding and prediction of nonlinear dynamics. Although discovery of governing equations based on observed system states (e.g., trajectory time series) has revealed success in a wide range of nonlinear dynamics, uncovering the closed-form equations directly from raw videos still remains an open challenge. To this end, we introduce a novel end-to-end unsupervised deep learning framework to uncover the mathematical structure of equations that governs the dynamics of moving objects in videos. Such an architecture consists of (1) an encoder-decoder network that learns low-dimensional spatial/pixel coordinates of the moving object, (2) a learnable Spatial-Physical Transformation component that creates mapping between the extracted spatial/pixel coordinates and the latent physical states of dynamics, and (3) a numerical integrator-based sparse regression module that uncovers the parsimonious closed-form governing equations of learned physical states and, meanwhile, serves as a constraint to the autoencoder. The efficacy of the proposed method is demonstrated by uncovering the governing equations of a variety of nonlinear dynamical systems depicted by moving objects in videos. The resulting computational framework enables discovery of parsimonious interpretable model in a flexible and accessible sensing environment where only videos are available.