# I-PHYRE: Interactive Physical Reasoning

arXiv: https://arxiv.org/abs/2312.03009 · HTML: https://mlsanity.com/paper/2312.03009/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [Learning Physical Intuition of Block Towers by Example](https://mlsanity.com/paper/1603.01312/) (2016)
- [ComPhy: Compositional Physical Reasoning of Objects and Events from Videos](https://mlsanity.com/paper/2205.01089/) (2022)
- [Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense](https://mlsanity.com/paper/2004.09044/) (2020)
- [Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning](https://mlsanity.com/paper/1907.09620/) (2019)
- [Learning compositional models of robot skills for task and motion planning](https://mlsanity.com/paper/2006.06444/) (2020)
- [Learning Long-term Visual Dynamics with Region Proposal Interaction Networks](https://mlsanity.com/paper/2008.02265/) (2020)
- [Generalization to New Actions in Reinforcement Learning](https://mlsanity.com/paper/2011.01928/) (2020)
- [A Survey on Machine Learning Approaches for Modelling Intuitive Physics](https://mlsanity.com/paper/2202.06481/) (2022)
- [Forward Prediction for Physical Reasoning](https://mlsanity.com/paper/2006.10734/) (2020)
- [HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving](https://mlsanity.com/paper/2102.11344/) (2021)
- [On the Learning Mechanisms in Physical Reasoning](https://mlsanity.com/paper/2210.02075/) (2022)
- [A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories](https://mlsanity.com/paper/2111.08826/) (2021)
- [Describing Physics For Physical Reasoning: Force-based Sequential Manipulation Planning](https://mlsanity.com/paper/2002.12780/) (2020)
- [Understanding Physical Effects for Effective Tool-use](https://mlsanity.com/paper/2206.14998/) (2022)

## Built on by (most cited first, 15 of 15)

- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [Learning Interactive World Model for Object-Centric Reinforcement Learning](https://mlsanity.com/paper/2511.02225/) (2025)
- [VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction](https://mlsanity.com/paper/2602.13294/) (2026)
- [Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields](https://mlsanity.com/paper/2602.00148/) (2026)
- [MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics](https://mlsanity.com/paper/2606.01538/) (2026)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [Hierarchical Experimentalist Agents](https://mlsanity.com/paper/2606.29315/) (2026)
- [Principia: Relational Physics Tests for Video Models](https://mlsanity.com/paper/2609.04200/) (2026)
- [Neural Force Field: Few-shot Learning of Generalized Physical Reasoning](https://mlsanity.com/paper/2502.08987/) (2025)
- [IMBench: A Benchmark for Intuitive Robotic Manipulation](https://mlsanity.com/paper/2607.15641/) (2026)
- [Moving Out: Physically-grounded Human-AI Collaboration](https://mlsanity.com/paper/2507.18623/) (2025)
- [XDen-1K: A Density Field Dataset of Real-World Objects](https://mlsanity.com/paper/2512.10668/) (2025)
- [Overhang Tower: Resource-Rational Adaptation in Sequential Physical Planning](https://mlsanity.com/paper/2604.09072/) (2026)
- [CompAdapt: Adaptable Composite Motion Modeling for Physics-Consistent Text-to-Video Generation](https://mlsanity.com/paper/2609.21455/) (2026)

## Abstract (arXiv)

Current evaluation protocols predominantly assess physical reasoning in stationary scenes, creating a gap in evaluating agents' abilities to interact with dynamic events. While contemporary methods allow agents to modify initial scene configurations and observe consequences, they lack the capability to interact with events in real time. To address this, we introduce I-PHYRE, a framework that challenges agents to simultaneously exhibit intuitive physical reasoning, multi-step planning, and in-situ intervention. Here, intuitive physical reasoning refers to a quick, approximate understanding of physics to address complex problems; multi-step denotes the need for extensive sequence planning in I-PHYRE, considering each intervention can significantly alter subsequent choices; and in-situ implies the necessity for timely object manipulation within a scene, where minor timing deviations can result in task failure. We formulate four game splits to scrutinize agents' learning and generalization of essential principles of interactive physical reasoning, fostering learning through interaction with representative scenarios. Our exploration involves three planning strategies and examines several supervised and reinforcement agents' zero-shot generalization proficiency on I-PHYRE. The outcomes highlight a notable gap between existing learning algorithms and human performance, emphasizing the imperative for more research in enhancing agents with interactive physical reasoning capabilities. The environment and baselines will be made publicly available.