# Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images

arXiv: https://arxiv.org/abs/1511.04048 · HTML: https://mlsanity.com/paper/1511.04048/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2015.

## Built on by (most cited first, 26 of 26)

- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [A Survey on Machine Learning Approaches for Modelling Intuitive Physics](https://mlsanity.com/paper/2202.06481/) (2022)
- [Pushing it out of the Way: Interactive Visual Navigation](https://mlsanity.com/paper/2104.14040/) (2021)
- [Understanding Physical Dynamics with Counterfactual World Modeling](https://mlsanity.com/paper/2312.06721/) (2023)
- [Knowledge-augmented Deep Learning and Its Applications: A Survey](https://mlsanity.com/paper/2212.00017/) (2022)
- [FORCE: Physics-aware Human-object Interaction](https://mlsanity.com/paper/2403.11237/) (2024)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Object and Relation Centric Representations for Push Effect Prediction](https://mlsanity.com/paper/2102.02100/) (2021)
- [ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos](https://mlsanity.com/paper/2601.05237/) (2026)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [Distilling Governing Laws and Source Input for Dynamical Systems from Videos](https://mlsanity.com/paper/2205.01314/) (2022)
- [Where Is The Ball: 3D Ball Trajectory Estimation From 2D Monocular Tracking](https://mlsanity.com/paper/2506.05763/) (2025)
- [InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning](https://mlsanity.com/paper/2509.12263/) (2025)
- [Envisioning the Future, One Step at a Time](https://mlsanity.com/paper/2604.09527/) (2026)
- [Dominant motion identification of multi-particle system using deep learning from video](https://mlsanity.com/paper/2104.12722/) (2021)
- [Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videos](https://mlsanity.com/paper/2106.04776/) (2021)
- [Recognising Affordances in Predicted Futures to Plan with Consideration of Non-canonical Affordance Effects](https://mlsanity.com/paper/2206.10920/) (2022)
- [T3VIP: Transformation-based 3D Video Prediction](https://mlsanity.com/paper/2209.11693/) (2022)
- [Adaptive Volumetric Mechanical Property Fields Invariant to Resolution](https://mlsanity.com/paper/2606.18231/) (2026)
- [MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction](https://mlsanity.com/paper/2607.16192/) (2026)
- [Interpretable Representation Learning from Videos using Nonlinear Priors](https://mlsanity.com/paper/2410.18539/) (2024)
- [Modeling The Object Representations Underlying Human Physical Reasoning](https://mlsanity.com/paper/2602.12486/) (2026)
- [PIVOT: Physics-Grounded Verification for AI-Generated Audio-Video Detection](https://mlsanity.com/paper/2609.15562/) (2026)
- [SnapPhysics: A Physics-Aware Scene Graph from a Single View for Interactive Mixed Reality Scenes](https://mlsanity.com/paper/2609.19815/) (2026)

## Abstract (arXiv)

In this paper, we study the challenging problem of predicting the dynamics of objects in static images. Given a query object in an image, our goal is to provide a physical understanding of the object in terms of the forces acting upon it and its long term motion as response to those forces. Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging. We define intermediate physical abstractions called Newtonian scenarios and introduce Newtonian Neural Network (N3) that learns to map a single image to a state in a Newtonian scenario. Our experimental evaluations show that our method can reliably predict dynamics of a query object from a single image. In addition, our approach can provide physical reasoning that supports the predicted dynamics in terms of velocity and force vectors. To spur research in this direction we compiled Visual Newtonian Dynamics (VIND) dataset that includes 6806 videos aligned with Newtonian scenarios represented using game engines, and 4516 still images with their ground truth dynamics.