ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Reacting to sudden physical hazards (catching a slipping plate, dodging a falling knife) is both a meaningful test of embodied intelligence and a hard requirement for deploying multimodal large language models (MLLMs) as the decision coreof household robots. Existing evaluations, however, probe intuitive physics passively through question answering over videos, or target deliberate, long-horizon tasks such as navigation and rearrangement; none measure whether a model can turn physical understanding into immediate, safety-critical action. We introduce ReactHuman, the first physics-grounded benchmark for human-like reactive decision-making, in which the evaluated MLLM acts as the brain of a simulated humanoid facing sudden household hazards; it spans 17 event families and over 1,000 bit-for-bit reproducible scenes with exact, annotation-free ground truth derived from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics (a foam anvil, a steel apple). We further design a five-metric suite that scores each reaction along three axes: reasonable, safe, and physically grounded. We physically execute every committed plan so that decisions have observable consequences. With this harness we evaluate seven representative MLLMs. Results show that reactive safety is far from solved: models mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale even when the chosen action is correct; none of these failures shrink with model scale. ReactHuman thus offers both a fine-grained diagnosis and a scalable training signal toward physically grounded, safety-aware embodied agents. The benchmark can be found here: https://huggingface.co/datasets/Alan123/reacthuman-benchmark-scaled
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 26 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
19 of 26 references.
Closest ancestors · 3
Same direction, newest first
- 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding27 Jan
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
- 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
Other references · 16
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
- 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
- 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
- 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
- 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
- 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
- 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
- 2025Do generative video models understand physical principles?cited by 161
4 more
- 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
- 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
- 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
- 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 54
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
- 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
- 2024A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment
- 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
- 2026Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
- 2024HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments
- 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
- 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Models