# Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Body

arXiv: https://arxiv.org/abs/2209.11355 · HTML: https://mlsanity.com/paper/2209.11355/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control](https://mlsanity.com/paper/1909.12077/) (2019)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Linearly-Recurrent Autoencoder Networks for Learning Dynamics](https://mlsanity.com/paper/1712.01378/) (2017)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints](https://mlsanity.com/paper/2010.13581/) (2020)
- [An Analysis of SVD for Deep Rotation Estimation](https://mlsanity.com/paper/2006.14616/) (2020)
- [Dissipative SymODEN: Encoding Hamiltonian Dynamics with Dissipation and Control into Deep Learning](https://mlsanity.com/paper/2002.08860/) (2020)
- [Deep Regression on Manifolds: A 3D Rotation Case Study](https://mlsanity.com/paper/2103.16317/) (2021)
- [Explorations in Homeomorphic Variational Auto-Encoding](https://mlsanity.com/paper/1807.04689/) (2018)
- [Visual Grounding of Learned Physical Models](https://mlsanity.com/paper/2004.13664/) (2020)
- [SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks](https://mlsanity.com/paper/1606.02378/) (2016)
- [Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control](https://mlsanity.com/paper/2007.01926/) (2020)
- [Hamiltonian-based Neural ODE Networks on the SE(3) Manifold For Dynamics Learning and Control](https://mlsanity.com/paper/2106.12782/) (2021)
- [A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty](https://mlsanity.com/paper/2006.01031/) (2020)
- [LagNetViP: A Lagrangian Neural Network for Video Prediction](https://mlsanity.com/paper/2010.12932/) (2020)
- [Learning Dynamical Systems with Side Information](https://mlsanity.com/paper/2008.10135/) (2020)
- [A General Framework for Structured Learning of Mechanical Systems](https://mlsanity.com/paper/1902.08705/) (2019)

## Built on by (most cited first, 4 of 4)

- [Hamiltonian Neural Networks with Automatic Symmetry Detection](https://mlsanity.com/paper/2301.07928/) (2023)
- [Behavior-Inspired Neural Networks for Relational Inference](https://mlsanity.com/paper/2406.14746/) (2024)
- [Discovering Phase Space Structure in Learned Hamiltonian Systems](https://mlsanity.com/paper/2509.23003/) (2025)
- [Frequency-Separable Hamiltonian Neural Network for Multi-Timescale Dynamics](https://mlsanity.com/paper/2603.06354/) (2026)

## Abstract (arXiv)

In many real-world settings, image observations of freely rotating 3D rigid bodies, such as satellites, may be available when low-dimensional measurements are not. However, the high-dimensionality of image data precludes the use of classical estimation techniques to learn the dynamics and a lack of interpretability reduces the usefulness of standard deep learning methods. In this work, we present a physics-informed neural network model to estimate and predict 3D rotational dynamics from image sequences. We achieve this using a multi-stage prediction pipeline that maps individual images to a latent representation homeomorphic to SO(3), computes angular velocities from latent pairs, and predicts future latent states using the Hamiltonian equations of motion with a learned representation of the Hamiltonian. We demonstrate the efficacy of our approach on a new rotating rigid-body dataset with sequences of rotating cubes and rectangular prisms with uniform and non-uniform density.