mlsanityData to 7 Oct 2026

PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding

arXiv 2501.16411 ↗First posted 27 Jan 2025cs.CV cs.AI cs.CLcited by 147

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world -- likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors. To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4% improvement on GPT-4o. Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 232 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

17 of 232 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Models26 Feb

Other references · 16

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2024Depth Anything V2cited by 1,385
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  6. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  9. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  10. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  11. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  12. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
4 more
  1. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 120
  2. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  3. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  4. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Taskcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 59; 2026: 88.

147 papers.

Sort:

By area:

  1. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  4. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  5. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  6. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 52
  7. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 44
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  10. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  11. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  12. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
38 more
  1. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  2. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  3. 2026Large Language Model Reasoning Failurescited by 20
  4. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  5. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 17
  6. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibilitycited by 16
  7. 2025AI Sees Your Location, But With A Bias Toward The Wealthy Worldcited by 14
  8. 2025Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMscited by 14
  9. 2025EgoNormia: Benchmarking Physical Social Norm Understandingcited by 13
  10. 2026Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoningcited by 13
  11. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  12. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  13. 2025SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMscited by 12
  14. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  15. 2025EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editingcited by 12
  16. 2026DanceOPD: On-Policy Generative Field Distillationcited by 12
  17. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 11
  18. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  19. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
  20. 2026PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Modelscited by 10
  21. 2026VOID: Video Object and Interaction Deletioncited by 9
  22. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  23. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosiscited by 8
  24. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalizationcited by 8
  25. 2025PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assemblycited by 7
  26. 2025Robot Learning from Any Imagescited by 7
  27. 2025Abstract 3D Perception for Spatial Intelligence in Vision-Language Modelscited by 7
  28. 2025Seeing the Wind from a Falling Leafcited by 7
  29. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmarkcited by 7
  30. 2025AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Sciencecited by 6
  31. 2025Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoTcited by 6
  32. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoningcited by 6
  33. 2026RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulationcited by 6
  34. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 6
  35. 2025Inferring Dynamic Physical Properties from Video Foundation Modelscited by 5
  36. 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
  37. 2026Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fieldscited by 5
  38. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agents
  2. 2026PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models
  3. 2026PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
  4. 2025QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models
  5. 2025PhysiAgent: An Embodied Agent Framework in Physical World
  6. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  7. 2024Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models
  8. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models