# PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2504.16074 · HTML: https://mlsanity.com/paper/2504.16074/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [Skywork: A More Open Bilingual Foundation Model](https://mlsanity.com/paper/2310.19341/) (2023)
- [OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models](https://mlsanity.com/paper/2410.09671/) (2024)
- [Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad](https://mlsanity.com/paper/2503.21934/) (2025)
- [Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models](https://mlsanity.com/paper/2503.21380/) (2025)

## Built on by (most cited first, 30 of 61)

- [LLaDA2.0: Scaling Up Diffusion Language Models to 100B](https://mlsanity.com/paper/2512.15745/) (2025)
- [Ministral 3](https://mlsanity.com/paper/2601.08584/) (2026)
- [OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation](https://mlsanity.com/paper/2506.07977/) (2025)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model](https://mlsanity.com/paper/2510.18855/) (2025)
- [LLaDA2.1: Speeding Up Text Diffusion via Token Editing](https://mlsanity.com/paper/2602.08676/) (2026)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning](https://mlsanity.com/paper/2505.19099/) (2025)
- [Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark](https://mlsanity.com/paper/2509.26574/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems](https://mlsanity.com/paper/2507.04766/) (2025)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors](https://mlsanity.com/paper/2507.15550/) (2025)
- [General Modular Harness for LLM Agents in Multi-Turn Gaming Environments](https://mlsanity.com/paper/2507.11633/) (2025)
- [PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement](https://mlsanity.com/paper/2604.23580/) (2026)
- [PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions](https://mlsanity.com/paper/2505.15472/) (2025)
- [Can Theoretical Physics Research Benefit from Language Agents?](https://mlsanity.com/paper/2506.06214/) (2025)
- [SUPERChem: A Multimodal Reasoning Benchmark in Chemistry](https://mlsanity.com/paper/2512.01274/) (2025)
- [PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation](https://mlsanity.com/paper/2605.09636/) (2026)
- [KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation](https://mlsanity.com/paper/2505.14552/) (2025)
- [PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problems](https://mlsanity.com/paper/2508.00079/) (2025)
- [MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning](https://mlsanity.com/paper/2506.05523/) (2025)
- [Agentic Exploration of Physics Models](https://mlsanity.com/paper/2509.24978/) (2025)
- [Chain of Event-Centric Causal Thought for Physically Plausible Video Generation](https://mlsanity.com/paper/2603.09094/) (2026)
- [PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research](https://mlsanity.com/paper/2604.15411/) (2026)
- [On Path to Multimodal Historical Reasoning: HistBench and HistAgent](https://mlsanity.com/paper/2505.20246/) (2025)
- [SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMs](https://mlsanity.com/paper/2508.08742/) (2025)
- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization](https://mlsanity.com/paper/2602.07064/) (2026)
- [MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics](https://mlsanity.com/paper/2606.01538/) (2026)

## Abstract (arXiv)

Current benchmarks for evaluating the reasoning capabilities of Large Language Models (LLMs) face significant limitations: task oversimplification, data contamination, and flawed evaluation items. These deficiencies necessitate more rigorous assessment methods. To address these limitations, we introduce PHYBench, a benchmark of 500 original physics problems ranging from high school to Physics Olympiad difficulty. PHYBench addresses data contamination through original content and employs a systematic curation pipeline to eliminate flawed items. Evaluations show that PHYBench activates more tokens and provides stronger differentiation between reasoning models compared to other baselines like AIME 2024, OlympiadBench and GPQA. Even the best-performing model, Gemini 2.5 Pro, achieves only 36.9% accuracy compared to human experts' 61.9%. To further enhance evaluation precision, we introduce the Expression Edit Distance (EED) Score for mathematical expression assessment, which improves sample efficiency by 204% over binary scoring. Moreover, PHYBench effectively elicits multi-step and multi-condition reasoning, providing a platform for examining models' reasoning robustness, preferences, and deficiencies. The benchmark results and dataset are publicly available at https://www.phybench.cn/.