# Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems

arXiv: https://arxiv.org/abs/2410.01376 · HTML: https://mlsanity.com/paper/2410.01376/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 54 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Discovering physical concepts with neural networks](https://mlsanity.com/paper/1807.10300/) (2018)
- [Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems](https://mlsanity.com/paper/2107.12146/) (2021)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling](https://mlsanity.com/paper/2102.13156/) (2021)
- [Learning Visual Predictive Models of Physics for Playing Billiards](https://mlsanity.com/paper/1511.07404/) (2015)
- [Zero-Shot Day-Night Domain Adaptation with a Physics Prior](https://mlsanity.com/paper/2108.05137/) (2021)
- [Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video](https://mlsanity.com/paper/1905.11169/) (2019)
- [Structured Object-Aware Physics Prediction for Video Modeling and Planning](https://mlsanity.com/paper/1910.02425/) (2019)
- [Robust Locally-Linear Controllable Embedding](https://mlsanity.com/paper/1710.05373/) (2017)
- [Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control](https://mlsanity.com/paper/2007.01926/) (2020)
- [Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling](https://mlsanity.com/paper/2109.05364/) (2021)
- [φ-SfT: Shape-from-Template with a Physics-Based Deformation Model](https://mlsanity.com/paper/2203.11938/) (2022)
- [Neural Implicit Representations for Physical Parameter Inference from a Single Video](https://mlsanity.com/paper/2204.14030/) (2022)
- [Deep Learning Models for Global Coordinate Transformations that Linearize PDEs](https://mlsanity.com/paper/1911.02710/) (2019)
- [NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spaces](https://mlsanity.com/paper/2006.01959/) (2020)
- [Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks](https://mlsanity.com/paper/1807.09244/) (2018)
- [Reasoning-Modulated Representations](https://mlsanity.com/paper/2107.08881/) (2021)
- [Sparse Surface Constraints for Combining Physics-based Elasticity Simulation and Correspondence-Free Object Reconstruction](https://mlsanity.com/paper/1910.01812/) (2019)
- [Learning to Identify Physical Parameters from Video Using Differentiable Physics](https://mlsanity.com/paper/2009.08292/) (2020)

## Built on by (most cited first, 13 of 13)

- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [Inferring Dynamic Physical Properties from Video Foundation Models](https://mlsanity.com/paper/2510.02311/) (2025)
- [SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation](https://mlsanity.com/paper/2512.03350/) (2025)
- [IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video](https://mlsanity.com/paper/2603.16432/) (2026)
- [Image2Garment: Simulation-ready Garment Generation from a Single Image](https://mlsanity.com/paper/2601.09658/) (2026)
- [Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos](https://mlsanity.com/paper/2605.20576/) (2026)
- [Principia: Relational Physics Tests for Video Models](https://mlsanity.com/paper/2609.04200/) (2026)
- [DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration](https://mlsanity.com/paper/2603.07545/) (2026)
- [Physics from Video: Identifiability of Time-Invariant Second-Order ODEs under Minimal Trajectory Conditions](https://mlsanity.com/paper/2606.00115/) (2026)
- [Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Data](https://mlsanity.com/paper/2604.02031/) (2026)
- [EMMA: Extracting Multiple physical parameters from Multimodal Data](https://mlsanity.com/paper/2605.24047/) (2026)
- [LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting](https://mlsanity.com/paper/2608.16324/) (2026)

## Abstract (arXiv)

Extracting physical dynamical system parameters from recorded observations is key in natural science. Current methods for automatic parameter estimation from video train supervised deep networks on large datasets. Such datasets require labels, which are difficult to acquire. While some unsupervised techniques--which depend on frame prediction--exist, they suffer from long training times, initialization instabilities, only consider motion-based dynamical systems, and are evaluated mainly on synthetic data. In this work, we propose an unsupervised method to estimate the physical parameters of known, continuous governing equations from single videos suitable for different dynamical systems beyond motion and robust to initialization. Moreover, we remove the need for frame prediction by implementing a KL-divergence-based loss function in the latent space, which avoids convergence to trivial solutions and reduces model size and compute. We first evaluate our model on synthetic data, as commonly done. After which, we take the field closer to reality by recording Delfys75: our own real-world dataset of 75 videos for five different types of dynamical systems to evaluate our method and others. Our method compares favorably to others. %, yet, and real-world video datasets and demonstrate improved parameter estimation accuracy compared to existing methods. Code and data are available online:https://github.com/Alejandro-neuro/Learning_physics_from_video.