mlsanityData to 14 Oct 2026

PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models

arXiv 2504.16074 ↗First posted 22 Apr 2025cs.CLcited by 61

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Current benchmarks for evaluating the reasoning capabilities of Large Language Models (LLMs) face significant limitations: task oversimplification, data contamination, and flawed evaluation items. These deficiencies necessitate more rigorous assessment methods. To address these limitations, we introduce PHYBench, a benchmark of 500 original physics problems ranging from high school to Physics Olympiad difficulty. PHYBench addresses data contamination through original content and employs a systematic curation pipeline to eliminate flawed items. Evaluations show that PHYBench activates more tokens and provides stronger differentiation between reasoning models compared to other baselines like AIME 2024, OlympiadBench and GPQA. Even the best-performing model, Gemini 2.5 Pro, achieves only 36.9% accuracy compared to human experts' 61.9%. To further enhance evaluation precision, we introduce the Expression Edit Distance (EED) Score for mathematical expression assessment, which improves sample efficiency by 204% over binary scoring. Moreover, PHYBench effectively elicits multi-step and multi-condition reasoning, providing a platform for examining models' reasoning robustness, preferences, and deficiencies. The benchmark results and dataset are publicly available at https://www.phybench.cn/.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 35 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

19 of 35 references.

References · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
7 more
  1. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  2. 2025Humanity's Last Examcited by 491
  3. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  4. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  5. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
  6. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
  7. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 30.

61 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  2. 2026Ministral 3cited by 114
  3. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 87
  4. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  5. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  6. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  7. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  8. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 34
  9. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  10. 2026Large Language Model Reasoning Failurescited by 21
  11. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 18
  12. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 16
38 more
  1. 2025PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priorscited by 15
  2. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 11
  3. 2026PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinementcited by 10
  4. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensionscited by 9
  5. 2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9
  6. 2025SUPERChem: A Multimodal Reasoning Benchmark in Chemistrycited by 9
  7. 2026PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generationcited by 8
  8. 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluationcited by 7
  9. 2025PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problemscited by 7
  10. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoningcited by 6
  11. 2025Agentic Exploration of Physics Modelscited by 6
  12. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  13. 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Researchcited by 6
  14. 2025On Path to Multimodal Historical Reasoning: HistBench and HistAgentcited by 5
  15. 2025SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMscited by 5
  16. 2025EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoningcited by 5
  17. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  18. 2026MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamicscited by 5
  19. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Modelscited by 4
  20. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoningcited by 4
  21. 2025CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researcherscited by 4
  22. 2026Vision Language Models Cannot Reason About Physical Transformationcited by 4
  23. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  24. 2025Iterated Agent for Symbolic Regressioncited by 3
  25. 2025Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signaturescited by 2
  26. 2025PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Researchcited by 2
  27. 2026Reward Design for Physical Reasoning in Vision-Language Modelscited by 2
  28. 2026MaD Physics: Evaluating information seeking under constraints in physical environmentscited by 2
  29. 2026Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarkscited by 2
  30. 2025Mastering Olympiad-Level Physics with Artificial Intelligencecited by 1
  31. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedbackcited by 1
  32. 2026How Emotion Shapes the Behavior of LLMs and Agents: A Mechanistic Studycited by 1
  33. 2026PolyReal: A Benchmark for Real-World Polymer Science Workflowscited by 1
  34. 2026Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMscited by 1
  35. 2026Heterogeneous Scientific Foundation Model Collaborationcited by 1
  36. 2026AxQM: A Textbook-Scale Benchmark for Formal Proof Synthesis in a Library of Finite-Dimensional Quantum Mechanicscited by 1
  37. 2025Language Models Might Not Understand You: Evaluating Theory of Mind via Story Promptingcited by 0
  38. 2025LOCA: Logical Chain Augmentation for Scientific Corpus Cleaningcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models
  2. 2026PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
  3. 2026PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems
  4. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
  5. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
  6. 2023ARB: Advanced Reasoning Benchmark for Large Language Models
  7. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models