# ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting

arXiv: https://arxiv.org/abs/2608.20009 · HTML: https://mlsanity.com/paper/2608.20009/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning](https://mlsanity.com/paper/2212.00259/) (2022)
- [Equivariant Graph Mechanics Networks with Constraints](https://mlsanity.com/paper/2203.06442/) (2022)
- [Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations](https://mlsanity.com/paper/2011.03880/) (2020)
- [Visual Grounding of Learned Physical Models](https://mlsanity.com/paper/2004.13664/) (2020)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties](https://mlsanity.com/paper/2306.15668/) (2023)
- [Physical Property Understanding from Language-Embedded Feature Fields](https://mlsanity.com/paper/2404.04242/) (2024)
- [Differentiable Physics Simulation of Dynamics-Augmented Neural Objects](https://mlsanity.com/paper/2210.09420/) (2022)
- [SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases](https://mlsanity.com/paper/2308.13212/) (2023)
- [Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression](https://mlsanity.com/paper/2501.03674/) (2025)
- [Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2502.12425/) (2025)
- [3D-Aware Visual Question Answering about Parts, Poses and Occlusions](https://mlsanity.com/paper/2310.17914/) (2023)
- [Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning](https://mlsanity.com/paper/2512.15153/) (2025)
- [Compositional Physical Reasoning of Objects and Events from Videos](https://mlsanity.com/paper/2408.02687/) (2024)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [Intrinsic Physical Concepts Discovery with Object-Centric Predictive Models](https://mlsanity.com/paper/2303.01869/) (2023)
- [PAINET: A Principled Efficient Transformer for 3D Dynamics Modeling](https://mlsanity.com/paper/2510.04233/) (2025)
- [Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction](https://mlsanity.com/paper/2603.10597/) (2026)
- [Learning Physical Dynamics for Object-centric Visual Prediction](https://mlsanity.com/paper/2403.10079/) (2024)

## Abstract (arXiv)

Understanding object dynamics requires not only predicting future trajectories but also examining whether a model captures the physical properties that govern motion. However, existing benchmarks rarely expose object-level physical properties as explicit evaluation targets alongside trajectory forecasting. To address this gap, we introduce ExPhy, a multi-object trajectory forecasting benchmark containing 24,000 simulated physical scenes with explicit object-level labels for mass, friction, and restitution. ExPhy provides observed and future trajectories together with an in-distribution (ID) split and two out-of-distribution (OOD) splits over physical parameters (OOD-Parameter) and initial states (OOD-Initial) for jointly evaluating trajectory forecasting and physical property estimation. We further instantiate textscPhyODE, a physics-guided model with an explicit property interface that estimates physical properties from observed trajectories and uses them for differentiable future rollout. On the long-horizon OOD-Initial setting, textscPhyODE reduces ADE and FDE by 33.1% and 31.0%, respectively, compared with the strongest baseline. Zero-shot evaluation on ComPhy further assesses cross-benchmark transfer. Property-level analyses reveal that accurate trajectory forecasting does not necessarily imply accurate recovery of the underlying physical properties. Code and data are available at https://github.com/Zest86/ExPhy.