# Which priors matter? Benchmarking models for learning latent dynamics

arXiv: https://arxiv.org/abs/2111.05458 · HTML: https://mlsanity.com/paper/2111.05458/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 69 references that are in the map.
- First posted on arXiv 9 Nov 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Unrolled Generative Adversarial Networks](https://mlsanity.com/paper/1611.02163/) (2016)
- [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020)
- [Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control](https://mlsanity.com/paper/1909.12077/) (2019)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [The Mechanics of n-Player Differentiable Games](https://mlsanity.com/paper/1802.05642/) (2018)
- [Hamiltonian Graph Networks with ODE Integrators](https://mlsanity.com/paper/1909.12790/) (2019)
- [The Numerics of GANs](https://mlsanity.com/paper/1705.10461/) (2017)
- [The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels](https://mlsanity.com/paper/2101.02722/) (2021)
- [Building machines that adapt and compute like brains](https://mlsanity.com/paper/1711.04203/) (2017)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Dissipative SymODEN: Encoding Hamiltonian Dynamics with Dissipation and Control into Deep Learning](https://mlsanity.com/paper/2002.08860/) (2020)
- [Benchmarking Energy-Conserving Neural Networks for Learning Dynamics from Data](https://mlsanity.com/paper/2012.02334/) (2020)
- [Equivariant Hamiltonian Flows](https://mlsanity.com/paper/1909.13739/) (2019)
- [Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control](https://mlsanity.com/paper/2007.01926/) (2020)
- [Variational Integrator Networks for Physically Structured Embeddings](https://mlsanity.com/paper/1910.09349/) (2019)
- [Nonseparable Symplectic Neural Networks](https://mlsanity.com/paper/2010.12636/) (2020)
- [LagNetViP: A Lagrangian Neural Network for Video Prediction](https://mlsanity.com/paper/2010.12932/) (2020)
- [Training Generative Adversarial Networks by Solving Ordinary Differential Equations](https://mlsanity.com/paper/2010.15040/) (2020)
- [Time-Reversal Symmetric ODE Network](https://mlsanity.com/paper/2007.11362/) (2020)
- [Forecasting Hamiltonian dynamics without canonical coordinates](https://mlsanity.com/paper/2010.15201/) (2020)

## Built on by (most cited first, 26 of 26)

- [PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2306.08827/) (2023)
- [Deconstructing the Inductive Biases of Hamiltonian Neural Networks](https://mlsanity.com/paper/2202.04836/) (2022)
- [FINDE: Neural Differential Equations for Finding and Preserving Invariant Quantities](https://mlsanity.com/paper/2210.00272/) (2022)
- [Knowledge-augmented Deep Learning and Its Applications: A Survey](https://mlsanity.com/paper/2212.00017/) (2022)
- [A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacoustic](https://mlsanity.com/paper/2204.06362/) (2022)
- [eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modeling](https://mlsanity.com/paper/2403.01371/) (2024)
- [Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Space](https://mlsanity.com/paper/2409.08439/) (2024)
- [A Riemannian Framework for Learning Reduced-order Lagrangian Dynamics](https://mlsanity.com/paper/2410.18868/) (2024)
- [Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets](https://mlsanity.com/paper/2303.17824/) (2023)
- [SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision](https://mlsanity.com/paper/2111.05986/) (2021)
- [Hamiltonian latent operators for content and motion disentanglement in image sequences](https://mlsanity.com/paper/2112.01641/) (2021)
- [Stability-Informed Initialization of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2311.15890/) (2023)
- [Learning Trajectories of Hamiltonian Systems with Neural Networks](https://mlsanity.com/paper/2204.05077/) (2022)
- [Hamiltonian GAN](https://mlsanity.com/paper/2308.11216/) (2023)
- [Improving Generalization via Uncertainty Driven Perturbations](https://mlsanity.com/paper/2202.05737/) (2022)
- [Gaussian Process Latent Variable Modeling for Few-shot Time Series Forecasting](https://mlsanity.com/paper/2212.10306/) (2022)
- [Learning Hamiltonian Dynamics at Scale: A Differential-Geometric Approach](https://mlsanity.com/paper/2509.24627/) (2025)
- [CD2: Fine-grained 3D Mesh Reconstruction With Twice Chamfer Distance](https://mlsanity.com/paper/2206.00447/) (2022)
- [Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers](https://mlsanity.com/paper/2306.12077/) (2023)
- [Poisson-Dirac Neural Networks for Modeling Coupled Dynamical Systems across Domains](https://mlsanity.com/paper/2410.11480/) (2024)
- [Learning Permutation-invariant Macroscopic Dynamics](https://mlsanity.com/paper/2605.30812/) (2026)
- [KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images](https://mlsanity.com/paper/2206.11030/) (2022)
- [Neural Langevin Dynamics: towards interpretable Neural Stochastic Differential Equations](https://mlsanity.com/paper/2211.09537/) (2022)
- [Unsupervised Learning of Hybrid Latent Dynamics: A Learn-to-Identify Framework](https://mlsanity.com/paper/2403.08194/) (2024)
- [Learning partially observed systems with neural Hamiltonian ordinary differential equations](https://mlsanity.com/paper/2605.23510/) (2026)
- [Conservation Buys Stability and Factoring Buys Counterfactuals in Physical World Models](https://mlsanity.com/paper/2609.19674/) (2026)

## Abstract (arXiv)

Learning dynamics is at the heart of many important applications of machine learning (ML), such as robotics and autonomous driving. In these settings, ML algorithms typically need to reason about a physical system using high dimensional observations, such as images, without access to the underlying state. Recently, several methods have proposed to integrate priors from classical mechanics into ML models to address the challenge of physical reasoning from images. In this work, we take a sober look at the current capabilities of these models. To this end, we introduce a suite consisting of 17 datasets with visual observations based on physical systems exhibiting a wide range of dynamics. We conduct a thorough and detailed comparison of the major classes of physically inspired methods alongside several strong baselines. While models that incorporate physical priors can often learn latent spaces with desirable properties, our results demonstrate that these methods fail to significantly improve upon standard techniques. Nonetheless, we find that the use of continuous and time-reversible dynamics benefits models of all classes.