# Learning to Identify Physical Parameters from Video Using Differentiable Physics

arXiv: https://arxiv.org/abs/2009.08292 · HTML: https://mlsanity.com/paper/2009.08292/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 12 of 12)

- [Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning](https://mlsanity.com/paper/2110.00792/) (2021)
- [φ-SfT: Shape-from-Template with a Physics-Based Deformation Model](https://mlsanity.com/paper/2203.11938/) (2022)
- [Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks](https://mlsanity.com/paper/2501.13329/) (2025)
- [Neural Implicit Representations for Physical Parameter Inference from a Single Video](https://mlsanity.com/paper/2204.14030/) (2022)
- [DiffSDFSim: Differentiable Rigid-Body Dynamics With Implicit Shapes](https://mlsanity.com/paper/2111.15318/) (2021)
- [Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems](https://mlsanity.com/paper/2410.01376/) (2024)
- [Vision-based system identification and 3D keypoint discovery using dynamics constraints](https://mlsanity.com/paper/2109.05928/) (2021)
- [Learning Object Manipulation Skills from Video via Approximate Differentiable Physics](https://mlsanity.com/paper/2208.01960/) (2022)
- [Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fields](https://mlsanity.com/paper/2503.19976/) (2025)
- [Physics from Video: Identifiability of Time-Invariant Second-Order ODEs under Minimal Trajectory Conditions](https://mlsanity.com/paper/2606.00115/) (2026)
- [EMMA: Extracting Multiple physical parameters from Multimodal Data](https://mlsanity.com/paper/2605.24047/) (2026)
- [PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction](https://mlsanity.com/paper/2610.17288/) (2026)

## Abstract (arXiv)

Video representation learning has recently attracted attention in computer vision due to its applications for activity and scene forecasting or vision-based planning and control. Video prediction models often learn a latent representation of video which is encoded from input frames and decoded back into images. Even when conditioned on actions, purely deep learning based architectures typically lack a physically interpretable latent space. In this study, we use a differentiable physics engine within an action-conditional video representation network to learn a physical latent representation. We propose supervised and self-supervised learning methods to train our network and identify physical properties. The latter uses spatial transformers to decode physical states back into images. The simulation scenarios in our experiments comprise pushing, sliding and colliding objects, for which we also analyze the observability of the physical properties. In experiments we demonstrate that our network can learn to encode images and identify physical properties like mass and friction from videos and action sequences in the simulated scenarios. We evaluate the accuracy of our supervised and self-supervised methods and compare it with a system identification baseline which directly learns from state trajectories. We also demonstrate the ability of our method to predict future video frames from input images and actions.