# ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems

arXiv: https://arxiv.org/abs/2507.04766 · HTML: https://mlsanity.com/paper/2507.04766/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 44 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning](https://mlsanity.com/paper/2007.08124/) (2020)
- [PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models](https://mlsanity.com/paper/2504.16074/) (2025)
- [UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models](https://mlsanity.com/paper/2502.00334/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics](https://mlsanity.com/paper/2502.15815/) (2025)
- [Scaling Laws for Fact Memorization of Large Language Models](https://mlsanity.com/paper/2406.15720/) (2024)
- [Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation](https://mlsanity.com/paper/2406.14644/) (2024)
- [PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving](https://mlsanity.com/paper/2504.10885/) (2025)
- [Scaling Physical Reasoning with the PHYSICS Dataset](https://mlsanity.com/paper/2506.00022/) (2025)

## Built on by (most cited first, 18 of 18)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [The Serial Scaling Hypothesis](https://mlsanity.com/paper/2507.12549/) (2025)
- [BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction](https://mlsanity.com/paper/2510.16559/) (2025)
- [General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks](https://mlsanity.com/paper/2604.11778/) (2026)
- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization](https://mlsanity.com/paper/2602.07064/) (2026)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [Training and Benchmarking Code Generation for Physics-Inspired Animations](https://mlsanity.com/paper/2602.10840/) (2026)
- [Mastering Olympiad-Level Physics with Artificial Intelligence](https://mlsanity.com/paper/2511.10515/) (2025)
- [Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMs](https://mlsanity.com/paper/2604.14188/) (2026)
- [Fine-Tuning Small Reasoning Models for Quantum Field Theory](https://mlsanity.com/paper/2604.18936/) (2026)
- [BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs](https://mlsanity.com/paper/2605.30900/) (2026)
- [LOCA: Logical Chain Augmentation for Scientific Corpus Cleaning](https://mlsanity.com/paper/2510.01249/) (2025)
- [Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation](https://mlsanity.com/paper/2510.24762/) (2025)
- [Testing Frontier Large Language Models' Physics Literacy in Parallel Physical Worlds](https://mlsanity.com/paper/2607.00276/) (2026)
- [Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents](https://mlsanity.com/paper/2607.14573/) (2026)
- [PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems](https://mlsanity.com/paper/2609.13152/) (2026)
- [Language models can notice an impossible engineering problem yet still report it as solved](https://mlsanity.com/paper/2610.06668/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown impressive performance in domains such as mathematics and programming, yet their capabilities in physics remain underexplored and poorly understood. Physics poses unique challenges that demand not only precise computation but also deep conceptual understanding and physical modeling skills. Existing benchmarks often fall short due to limited difficulty, multiple-choice formats, and static evaluation settings that fail to capture physical modeling ability. In this paper, we introduce ABench-Physics, a novel benchmark designed to rigorously evaluate LLMs' physical reasoning and generalization capabilities. ABench-Physics consists of two components: Phy_A, a static set of 400 graduate- or Olympiad-level problems; and Phy_B, a dynamic subset of 100 problems equipped with an automatic variation engine to test model robustness across changing conditions. All questions require precise numerical answers, with strict formatting and tolerance constraints. Our evaluation of several state-of-the-art LLMs reveals substantial performance gaps, highlighting persistent limitations in physical reasoning, especially in generalization to dynamic variants. ABench-Physics provides a challenging and diagnostic framework for advancing scientific reasoning in LLMs.