mlsanityData to 7 Oct 2026

PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving

arXiv 2503.21821 ↗First posted 26 Mar 2025cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

We introduce PHYSICS, a comprehensive benchmark for university-level physics problem solving. It contains 1297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. We develop a robust automated evaluation system for precise and reliable validation. Our evaluation of leading foundation models reveals substantial limitations. Even the most advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant challenges in solving high-level scientific problems. Through comprehensive error analysis, exploration of diverse prompting strategies, and Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify key areas for improvement, laying the foundation for future advancements.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

23 of 37 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
11 more
  1. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  3. 2025Humanity's Last Examcited by 467
  4. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  5. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  6. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  7. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  8. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  9. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  10. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  11. 2024HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematicscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 13.

31 papers.

Sort:

By area:

  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  2. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  3. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Examscited by 52
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  5. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 32
  6. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  7. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Sciencecited by 15
  8. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 14
  9. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensionscited by 8
  10. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  11. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  12. 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Researchcited by 5
19 more
  1. 2026OpenCompass: A Universal Evaluation Platform for Large Language Modelscited by 5
  2. 2025CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researcherscited by 4
  3. 2025Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Textscited by 2
  4. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learningcited by 2
  5. 2025DOoM: Difficult Olympiads of Mathcited by 1
  6. 2025PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Researchcited by 1
  7. 2026Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Researchcited by 1
  8. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  9. 2026Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMscited by 1
  10. 2026Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physicscited by 1
  11. 2026More Correct Mass, Worse Answers: Why Power Sampling Can Fail and How to Fix Itcited by 1
  12. 2026Decoupled Physical Modeling and Execution for Physics Reasoningcited by 1
  13. 2025LOCA: Logical Chain Augmentation for Scientific Corpus Cleaningcited by 0
  14. 2025Mastering Olympiad-Level Physics with Artificial Intelligencecited by 0
  15. 2025LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamicscited by 0
  16. 2026S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Servicescited by 0
  17. 2026FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoningcited by 0
  18. 2026Assessing AI in Introductory Physics Problem Solvingcited by 0
  19. 2026Training Specialist Models without Reasoning Trajectories for Domain Expert Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Physical Reasoning with the PHYSICS Dataset
  2. 2025Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving
  3. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
  4. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics
  5. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
  6. 2026How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
  7. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
  8. 2025PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning