mlsanityData to 7 Oct 2026

Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems

arXiv 2410.01376 ↗First posted 2 Oct 2024cs.CV physics.comp-phcited by 13

Not on the map. Why?

Abstract (arXiv)

Extracting physical dynamical system parameters from recorded observations is key in natural science. Current methods for automatic parameter estimation from video train supervised deep networks on large datasets. Such datasets require labels, which are difficult to acquire. While some unsupervised techniques--which depend on frame prediction--exist, they suffer from long training times, initialization instabilities, only consider motion-based dynamical systems, and are evaluated mainly on synthetic data. In this work, we propose an unsupervised method to estimate the physical parameters of known, continuous governing equations from single videos suitable for different dynamical systems beyond motion and robust to initialization. Moreover, we remove the need for frame prediction by implementing a KL-divergence-based loss function in the latent space, which avoids convergence to trivial solutions and reduces model size and compute. We first evaluate our model on synthetic data, as commonly done. After which, we take the field closer to reality by recording Delfys75: our own real-world dataset of 75 videos for five different types of dynamical systems to evaluate our method and others. Our method compares favorably to others. %, yet, and real-world video datasets and demonstrate improved parameter estimation accuracy compared to existing methods. Code and data are available online:https://github.com/Alejandro-neuro/Learning_physics_from_video.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

29 of 54 references.

References · 29

  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  2. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  3. 2020Lagrangian Neural Networkscited by 388
  4. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  5. 2020Universal Differential Equations for Scientific Machine Learningcited by 345
  6. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  7. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 205
  8. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  9. 2019Hamiltonian Generative Networkscited by 136
  10. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  11. 2018Discovering physical concepts with neural networkscited by 117
  12. 2021Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problemscited by 100
17 more
  1. 2017Visual Interaction Networkscited by 94
  2. 2021Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modelingcited by 59
  3. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  4. 2021Zero-Shot Day-Night Domain Adaptation with a Physics Priorcited by 47
  5. 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 43
  6. 2019Structured Object-Aware Physics Prediction for Video Modeling and Planningcited by 42
  7. 2020Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Controlcited by 38
  8. 2017Robust Locally-Linear Controllable Embeddingcited by 37
  9. 2021Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modelingcited by 27
  10. 2022φ-SfT: Shape-from-Template with a Physics-Based Deformation Modelcited by 23
  11. 2022Neural Implicit Representations for Physical Parameter Inference from a Single Videocited by 19
  12. 2019Deep Learning Models for Global Coordinate Transformations that Linearize PDEscited by 17
  13. 2020NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spacescited by 15
  14. 2018Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networkscited by 13
  15. 2021Reasoning-Modulated Representationscited by 13
  16. 2019Sparse Surface Constraints for Combining Physics-based Elasticity Simulation and Correspondence-Free Object Reconstructioncited by 12
  17. 2020Learning to Identify Physical Parameters from Video Using Differentiable Physicscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  2. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  3. 2025Inferring Dynamic Physical Properties from Video Foundation Modelscited by 5
  4. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  5. 2026Image2Garment: Simulation-ready Garment Generation from a Single Imagecited by 3
  6. 2026IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Videocited by 3
  7. 2026DreamSAC: Learning Hamiltonian World Models via Symmetry Explorationcited by 1
  8. 2026Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videoscited by 1
  9. 2026Physics from Video: Identifiability of Time-Invariant Second-Order ODEs under Minimal Trajectory Conditionscited by 1
  10. 2026Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Datacited by 0
  11. 2026EMMA: Extracting Multiple physical parameters from Multimodal Datacited by 0
  12. 2026LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splattingcited by 0
1 more
  1. 2026Principia: Relational Physics Tests for Video Modelscited by 0