mlsanityData to 7 Oct 2026

Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control

arXiv 2007.01926 ↗First posted 3 Jul 2020cs.LG cs.SY eess.SYcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent approaches for modelling dynamics of physical systems with neural networks enforce Lagrangian or Hamiltonian structure to improve prediction and generalization. However, when coordinates are embedded in high-dimensional data such as images, these approaches either lose interpretability or can only be applied to one particular example. We introduce a new unsupervised neural network model that learns Lagrangian dynamics from images, with interpretability that benefits prediction and control. The model infers Lagrangian dynamics on generalized coordinates that are simultaneously learned with a coordinate-aware variational autoencoder (VAE). The VAE is designed to account for the geometry of physical systems composed of multiple rigid bodies in the plane. By inferring interpretable Lagrangian dynamics, the model learns physical system properties, such as kinetic and potential energy, which enables long-term prediction of dynamics in the image space and synthesis of energy-based controllers.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 9; 2023: 5; 2024: 6; 2025: 3; 2026: 3.

38 papers.

Sort:

By area:

  1. 2021Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modelingcited by 40
  2. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  3. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  4. 2021Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Modelscited by 33
  5. 2023Physics-Informed Machine Learning for Modeling and Control of Dynamical Systemscited by 28
  6. 2021Which priors matter? Benchmarking models for learning latent dynamicscited by 26
  7. 2022Unravelling the Performance of Physics-informed Graph Neural Networks for Dynamical Systemscited by 23
  8. 2022Neural Implicit Representations for Physical Parameter Inference from a Single Videocited by 19
  9. 2021Applying Machine Learning to Study Fluid Mechanicscited by 13
  10. 2024Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systemscited by 13
  11. 2022Learning the Dynamics of Particle-based Systems with Lagrangian Graph Neural Networkscited by 11
  12. 2022Port-metriplectic neural networks: thermodynamics-informed machine learning of complex physical systemscited by 11
26 more
  1. 2022Thermodynamics of learning physical phenomenacited by 10
  2. 2024Generalized Lagrangian Neural Networkscited by 9
  3. 2023Learning to Predict 3D Rotational Dynamics from Images of a Rigid Body with Unknown Mass Distributioncited by 7
  4. 2024Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Spacecited by 7
  5. 2024A Riemannian Framework for Learning Reduced-order Lagrangian Dynamicscited by 7
  6. 2021Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitionscited by 6
  7. 2021Vision-based system identification and 3D keypoint discovery using dynamics constraintscited by 5
  8. 2021SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Visioncited by 5
  9. 2021Optimal Energy Shaping via Neural Approximatorscited by 4
  10. 2022Learning Trajectories of Hamiltonian Systems with Neural Networkscited by 4
  11. 2022ModLaNets: Learning Generalisable Dynamics via Modularity and Physical Inductive Biascited by 4
  12. 2022Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Bodycited by 4
  13. 2023Hamiltonian GANcited by 4
  14. 2024RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operatorcited by 4
  15. 2021Lagrangian Neural Network with Differentiable Symmetries and Relational Inductive Biascited by 3
  16. 2021Structure-Preserving Learning Using Gaussian Processes and Variational Integratorscited by 3
  17. 2023A Bayesian framework for discovering interpretable Lagrangian of dynamical systems from datacited by 2
  18. 2025Physically Plausible Multi-System Trajectory Generation and Symmetry Discoverycited by 2
  19. 2023Physics-Informed Kernel Embeddings: Integrating Prior System Knowledge with Data-Driven Controlcited by 1
  20. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimationcited by 1
  21. 2026Frequency-Separable Hamiltonian Neural Network for Multi-Timescale Dynamicscited by 1
  22. 2021Learn Proportional Derivative Controllable Latent Space from Pixelscited by 0
  23. 2022KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Imagescited by 0
  24. 2024Unsupervised Learning of Hybrid Latent Dynamics: A Learn-to-Identify Frameworkcited by 0
  25. 2026Co-Learning Port-Hamiltonian Systems and Optimal Energy-Shaping Controlcited by 0
  26. 2026Conservation Buys Stability and Factoring Buys Counterfactuals in Physical World Modelscited by 0