# LagNetViP: A Lagrangian Neural Network for Video Prediction

arXiv: https://arxiv.org/abs/2010.12932 · HTML: https://mlsanity.com/paper/2010.12932/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling](https://mlsanity.com/paper/2109.06407/) (2021)
- [Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models](https://mlsanity.com/paper/2102.06794/) (2021)
- [Which priors matter? Benchmarking models for learning latent dynamics](https://mlsanity.com/paper/2111.05458/) (2021)
- [Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing](https://mlsanity.com/paper/2211.07377/) (2022)
- [Combining Physics and Deep Learning to learn Continuous-Time Dynamics Models](https://mlsanity.com/paper/2110.01894/) (2021)
- [Port-metriplectic neural networks: thermodynamics-informed machine learning of complex physical systems](https://mlsanity.com/paper/2211.01873/) (2022)
- [Thermodynamics of learning physical phenomena](https://mlsanity.com/paper/2207.12749/) (2022)
- [Knowledge-augmented Deep Learning and Its Applications: A Survey](https://mlsanity.com/paper/2212.00017/) (2022)
- [How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.06335/) (2023)
- [Learning to Predict 3D Rotational Dynamics from Images of a Rigid Body with Unknown Mass Distribution](https://mlsanity.com/paper/2308.14666/) (2023)
- [SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision](https://mlsanity.com/paper/2111.05986/) (2021)
- [Discrete Lagrangian Neural Networks with Automatic Symmetry Discovery](https://mlsanity.com/paper/2211.10830/) (2022)
- [ModLaNets: Learning Generalisable Dynamics via Modularity and Physical Inductive Bias](https://mlsanity.com/paper/2206.12325/) (2022)
- [Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Body](https://mlsanity.com/paper/2209.11355/) (2022)
- [Hamiltonian GAN](https://mlsanity.com/paper/2308.11216/) (2023)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [Predictions Based on Pixel Data: Insights from PDEs and Finite Differences](https://mlsanity.com/paper/2305.00723/) (2023)
- [REPeat: A Real2Sim2Real Approach for Pre-acquisition of Soft Food Items in Robot-assisted Feeding](https://mlsanity.com/paper/2410.10017/) (2024)
- [Discovering Phase Space Structure in Learned Hamiltonian Systems](https://mlsanity.com/paper/2509.23003/) (2025)
- [Physics-Informed Kernel Embeddings: Integrating Prior System Knowledge with Data-Driven Control](https://mlsanity.com/paper/2301.03565/) (2023)
- [Do Two AI Scientists Agree?](https://mlsanity.com/paper/2504.02822/) (2025)
- [Frequency-Separable Hamiltonian Neural Network for Multi-Timescale Dynamics](https://mlsanity.com/paper/2603.06354/) (2026)
- [KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images](https://mlsanity.com/paper/2206.11030/) (2022)
- [Unsupervised Learning of Hybrid Latent Dynamics: A Learn-to-Identify Framework](https://mlsanity.com/paper/2403.08194/) (2024)

## Abstract (arXiv)

The dominant paradigms for video prediction rely on opaque transition models where neither the equations of motion nor the underlying physical quantities of the system are easily inferred. The equations of motion, as defined by Newton's second law, describe the time evolution of a physical system state and can therefore be applied toward the determination of future system states. In this paper, we introduce a video prediction model where the equations of motion are explicitly constructed from learned representations of the underlying physical quantities. To achieve this, we simultaneously learn a low-dimensional state representation and system Lagrangian. The kinetic and potential energy terms of the Lagrangian are distinctly modelled and the low-dimensional equations of motion are explicitly constructed using the Euler-Lagrange equations. We demonstrate the efficacy of this approach for video prediction on image sequences rendered in modified OpenAI gym Pendulum-v0 and Acrobot environments.