# Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control

arXiv: https://arxiv.org/abs/2007.01926 · HTML: https://mlsanity.com/paper/2007.01926/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 30 of 38)

- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling](https://mlsanity.com/paper/2109.06407/) (2021)
- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models](https://mlsanity.com/paper/2102.06794/) (2021)
- [Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems](https://mlsanity.com/paper/2306.13867/) (2023)
- [Which priors matter? Benchmarking models for learning latent dynamics](https://mlsanity.com/paper/2111.05458/) (2021)
- [Unravelling the Performance of Physics-informed Graph Neural Networks for Dynamical Systems](https://mlsanity.com/paper/2211.05520/) (2022)
- [Neural Implicit Representations for Physical Parameter Inference from a Single Video](https://mlsanity.com/paper/2204.14030/) (2022)
- [Applying Machine Learning to Study Fluid Mechanics](https://mlsanity.com/paper/2110.02083/) (2021)
- [Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systems](https://mlsanity.com/paper/2410.01376/) (2024)
- [Learning the Dynamics of Particle-based Systems with Lagrangian Graph Neural Networks](https://mlsanity.com/paper/2209.01476/) (2022)
- [Port-metriplectic neural networks: thermodynamics-informed machine learning of complex physical systems](https://mlsanity.com/paper/2211.01873/) (2022)
- [Thermodynamics of learning physical phenomena](https://mlsanity.com/paper/2207.12749/) (2022)
- [Generalized Lagrangian Neural Networks](https://mlsanity.com/paper/2401.03728/) (2024)
- [Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitions](https://mlsanity.com/paper/2106.04165/) (2021)
- [Learning to Predict 3D Rotational Dynamics from Images of a Rigid Body with Unknown Mass Distribution](https://mlsanity.com/paper/2308.14666/) (2023)
- [Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Space](https://mlsanity.com/paper/2409.08439/) (2024)
- [A Riemannian Framework for Learning Reduced-order Lagrangian Dynamics](https://mlsanity.com/paper/2410.18868/) (2024)
- [Vision-based system identification and 3D keypoint discovery using dynamics constraints](https://mlsanity.com/paper/2109.05928/) (2021)
- [SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision](https://mlsanity.com/paper/2111.05986/) (2021)
- [Optimal Energy Shaping via Neural Approximators](https://mlsanity.com/paper/2101.05537/) (2021)
- [Lagrangian Neural Network with Differentiable Symmetries and Relational Inductive Bias](https://mlsanity.com/paper/2110.03266/) (2021)
- [Learning Trajectories of Hamiltonian Systems with Neural Networks](https://mlsanity.com/paper/2204.05077/) (2022)
- [ModLaNets: Learning Generalisable Dynamics via Modularity and Physical Inductive Bias](https://mlsanity.com/paper/2206.12325/) (2022)
- [Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Body](https://mlsanity.com/paper/2209.11355/) (2022)
- [Hamiltonian GAN](https://mlsanity.com/paper/2308.11216/) (2023)
- [RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operator](https://mlsanity.com/paper/2409.03107/) (2024)
- [Structure-Preserving Learning Using Gaussian Processes and Variational Integrators](https://mlsanity.com/paper/2112.05451/) (2021)
- [A Bayesian framework for discovering interpretable Lagrangian of dynamical systems from data](https://mlsanity.com/paper/2310.06241/) (2023)
- [Discovering Phase Space Structure in Learned Hamiltonian Systems](https://mlsanity.com/paper/2509.23003/) (2025)

## Abstract (arXiv)

Recent approaches for modelling dynamics of physical systems with neural networks enforce Lagrangian or Hamiltonian structure to improve prediction and generalization. However, when coordinates are embedded in high-dimensional data such as images, these approaches either lose interpretability or can only be applied to one particular example. We introduce a new unsupervised neural network model that learns Lagrangian dynamics from images, with interpretability that benefits prediction and control. The model infers Lagrangian dynamics on generalized coordinates that are simultaneously learned with a coordinate-aware variational autoencoder (VAE). The VAE is designed to account for the geometry of physical systems composed of multiple rigid bodies in the plane. By inferring interpretable Lagrangian dynamics, the model learns physical system properties, such as kinetic and potential energy, which enables long-term prediction of dynamics in the image space and synthesis of energy-based controllers.