mlsanityData to 7 Oct 2026

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

arXiv 2502.00334 ↗First posted 1 Feb 2025cs.CL cs.AIcited by 45

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics, underscoring the need for a comprehensive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and comprehensive benchmark specifically designed to evaluate UnderGraduate-level Physics (UGPhysics) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-level physics problems in both English and Chinese, covering 13 subjects with seven different answer types and four distinct physics reasoning skills, all rigorously screened for data leakage. Additionally, we develop a Model-Assistant Rule-based Judgment (MARJ) pipeline specifically tailored for assessing answer correctness of physics problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAI-o1-mini), emphasizes the necessity for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements in AI for physics reasoning. Codes and data are available at https://github.com/YangLabHKUST/UGPhysics .

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 77 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

45 of 77 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models18 Dec
  2. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents1 Dec
  3. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?10 Oct

Other references · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
30 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  5. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  8. 2023Llemma: An Open Language Model For Mathematicscited by 385
  9. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  10. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  11. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  12. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  13. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  14. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  15. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  16. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  17. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  18. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  19. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  20. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  21. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  22. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 76
  23. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  24. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  25. 2024VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbationcited by 29
  26. 2024Can LLMs Solve longer Math Word Problems Better?cited by 22
  27. 2023Using Large Language Model to Solve and Explain Physics Word Problems Approaching Human Levelcited by 15
  28. 2024Can We Verify Step by Step for Incorrect Answer Detection?cited by 15
  29. 2024E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Modelscited by 13
  30. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 25.

45 papers.

Sort:

By area:

  1. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  3. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 32
  4. 2026Large Language Model Reasoning Failurescited by 20
  5. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Modelscited by 18
  6. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 17
  7. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 16
  8. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 14
  9. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 14
  10. 2025DeepPHY: Benchmarking Agentic VLMs on Physical Reasoningcited by 13
  11. 2026Computer Environments Elicit General Agentic Intelligence in LLMscited by 11
  12. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensionscited by 8
33 more
  1. 2025PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problemscited by 7
  2. 2026Rethinking Privileged Information in On-Policy Self-Distillationcited by 6
  3. 2025MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Taskcited by 5
  4. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasonerscited by 5
  5. 2025Lean4Physics: Comprehensive Reasoning Framework for College-level Physics in Lean4cited by 5
  6. 2026FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoningcited by 5
  7. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 5
  8. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Modelscited by 4
  9. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoningcited by 4
  10. 2025EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoningcited by 4
  11. 2026WildSci: Advancing Scientific Reasoning from In-the-Wild Literaturecited by 4
  12. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  13. 2026Symbol-Equivariant Recurrent Reasoning Modelscited by 3
  14. 2026Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraintscited by 2
  15. 2026Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Trainingcited by 2
  16. 2025Superstudent intelligence in thermodynamicscited by 1
  17. 2025Spectral Logit Sculpting: Adaptive Low-Rank Logit Transformation for Controlled Text Generationcited by 1
  18. 2026Toward Evaluation Frameworks for Multi-Agent Scientific AI Systemscited by 1
  19. 2026Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMscited by 1
  20. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  21. 2026Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physicscited by 1
  22. 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMscited by 1
  23. 2025Mastering Olympiad-Level Physics with Artificial Intelligencecited by 0
  24. 2026Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervisioncited by 0
  25. 2026Infinite Problem Generator: Verifiably Scaling Physics Reasoning Data with Agentic Workflowscited by 0
  26. 2026Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gaincited by 0
  27. 2026Physics-R1: An Audited Olympiad Corpus and Released Verifiers for Visual Physics Reasoningcited by 0
  28. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  29. 2026ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?cited by 0
  30. 2026Position: The Systemic Lack of Agency in Visual Reasoningcited by 0
  31. 2026AxQM: A Textbook-Scale Benchmark for Formal Proof Synthesis in a Library of Finite-Dimensional Quantum Mechanicscited by 0
  32. 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistantscited by 0
  33. 2026How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Physics: A Comprehensive Benchmark for Multimodal LLMs Reasoning on Chinese Multi-Subject Physics Problems
  2. 2026PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
  3. 2025Scaling Physical Reasoning with the PHYSICS Dataset
  4. 2025Symbolic or Numerical? Understanding Physics Problem Solving in Reasoning LLMs
  5. 2025Dissecting Physics Reasoning in Small Language Models: A Multi-Dimensional Analysis from an Educational Perspective
  6. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
  7. 2026OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora
  8. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment