# QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models

arXiv: https://arxiv.org/abs/2512.19526 · HTML: https://mlsanity.com/paper/2512.19526/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 22 Dec 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning](https://mlsanity.com/paper/2503.15558/) (2025)
- [VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation](https://mlsanity.com/paper/2503.06800/) (2025)
- [Cambrian-S: Towards Spatial Supersensing in Video](https://mlsanity.com/paper/2511.04670/) (2025)
- [Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection](https://mlsanity.com/paper/2405.10300/) (2024)
- [Holistic Evaluation of Multimodal LLMs on Spatial Intelligence](https://mlsanity.com/paper/2508.13142/) (2025)
- [ContPhy: Continuum Physical Concept Learning and Reasoning from Videos](https://mlsanity.com/paper/2402.06119/) (2024)
- [DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning](https://mlsanity.com/paper/2508.05405/) (2025)
- [Wild2Avatar: Rendering Humans Behind Occlusions](https://mlsanity.com/paper/2401.00431/) (2023)

## Built on by (most cited first, 20 of 20)

- [Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning](https://mlsanity.com/paper/2608.27549/) (2026)
- [OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization](https://mlsanity.com/paper/2602.07064/) (2026)
- [Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling](https://mlsanity.com/paper/2602.08058/) (2026)
- [PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning](https://mlsanity.com/paper/2608.04575/) (2026)
- [Agentic Physical AI toward a Domain-Specific Foundation Model for Energy Systems: A Case Study on Nuclear Reactor Control](https://mlsanity.com/paper/2512.23292/) (2025)
- [4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes](https://mlsanity.com/paper/2610.03715/) (2026)
- [EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2604.22851/) (2026)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning](https://mlsanity.com/paper/2605.09266/) (2026)
- [Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts](https://mlsanity.com/paper/2605.29283/) (2026)
- [BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs](https://mlsanity.com/paper/2605.30900/) (2026)
- [A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models](https://mlsanity.com/paper/2606.28757/) (2026)
- [Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence](https://mlsanity.com/paper/2607.16401/) (2026)
- [Physics-Aware Video Generation via Agentic Planning and Graph-Guided Optimization](https://mlsanity.com/paper/2609.13006/) (2026)
- [NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics](https://mlsanity.com/paper/2605.08452/) (2026)
- [UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning](https://mlsanity.com/paper/2606.05576/) (2026)
- [ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?](https://mlsanity.com/paper/2606.07962/) (2026)
- [One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation](https://mlsanity.com/paper/2609.06190/) (2026)
- [OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalities](https://mlsanity.com/paper/2609.25738/) (2026)
- [Asking the World: Generalist Physical Reasoning through Agentic World Modeling and Probing](https://mlsanity.com/paper/2609.39135/) (2026)

## Abstract (arXiv)

Understanding the physical world is essential for generalist AI agents. However, it remains unclear whether state-of-the-art vision perception models (e.g., large VLMs) can reason physical properties quantitatively. Existing evaluations are predominantly VQA-based and qualitative, offering limited insight into whether these models can infer the kinematic quantities of moving objects from video observations. To address this, we present QuantiPhy, the first benchmark designed to quantitatively measure a VLM's physical reasoning ability. Comprising more than 3.3K video-text instances with numerical ground truth, QuantiPhy evaluates a VLM's performance on estimating an object's size, velocity, and acceleration at a given timestamp, using one of these properties as an input prior. The benchmark standardizes prompts and scoring to assess numerical accuracy, enabling fair comparisons across models. Our experiments on state-of-the-art VLMs reveal a consistent gap between their qualitative plausibility and actual numerical correctness. We further provide an in-depth analysis of key factors like background noise, counterfactual priors, and strategic prompting and find that state-of-the-art VLMs lean heavily on pre-trained world knowledge rather than faithfully using the provided visual and textual inputs as references when reasoning kinematic properties quantitatively. QuantiPhy offers the first rigorous, scalable testbed to move VLMs beyond mere verbal plausibility toward a numerically grounded physical understanding.