# Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

arXiv: https://arxiv.org/abs/2503.21821 · HTML: https://mlsanity.com/paper/2503.21821/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)
- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics](https://mlsanity.com/paper/2410.09988/) (2024)

## Built on by (most cited first, 30 of 31)

- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation](https://mlsanity.com/paper/2510.06303/) (2025)
- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning](https://mlsanity.com/paper/2505.19099/) (2025)
- [When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research](https://mlsanity.com/paper/2505.11855/) (2025)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions](https://mlsanity.com/paper/2505.15472/) (2025)
- [Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment](https://mlsanity.com/paper/2505.09438/) (2025)
- [Self-Improvement in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2510.02665/) (2025)
- [PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research](https://mlsanity.com/paper/2604.15411/) (2026)
- [OpenCompass: A Universal Evaluation Platform for Large Language Models](https://mlsanity.com/paper/2605.19276/) (2026)
- [CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers](https://mlsanity.com/paper/2510.05228/) (2025)
- [Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Texts](https://mlsanity.com/paper/2505.17950/) (2025)
- [PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Research](https://mlsanity.com/paper/2512.19799/) (2025)
- [DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning](https://mlsanity.com/paper/2602.11089/) (2026)
- [Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research](https://mlsanity.com/paper/2602.15889/) (2026)
- [Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs](https://mlsanity.com/paper/2605.06225/) (2026)
- [DOoM: Difficult Olympiads of Math](https://mlsanity.com/paper/2509.23529/) (2025)
- [Mastering Olympiad-Level Physics with Artificial Intelligence](https://mlsanity.com/paper/2511.10515/) (2025)
- [Fine-Tuning Small Reasoning Models for Quantum Field Theory](https://mlsanity.com/paper/2604.18936/) (2026)
- [Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physics](https://mlsanity.com/paper/2605.17104/) (2026)
- [More Correct Mass, Worse Answers: Why Power Sampling Can Fail and How to Fix It](https://mlsanity.com/paper/2608.14420/) (2026)
- [Decoupled Physical Modeling and Execution for Physics Reasoning](https://mlsanity.com/paper/2608.22126/) (2026)
- [LOCA: Logical Chain Augmentation for Scientific Corpus Cleaning](https://mlsanity.com/paper/2510.01249/) (2025)
- [LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics](https://mlsanity.com/paper/2512.21010/) (2025)
- [S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services](https://mlsanity.com/paper/2602.14017/) (2026)
- [FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning](https://mlsanity.com/paper/2604.03893/) (2026)
- [Assessing AI in Introductory Physics Problem Solving](https://mlsanity.com/paper/2607.14303/) (2026)

## Abstract (arXiv)

We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended problems from PhD-qualifying exams across six core physics domains. Solving these problems requires applying domain-specific knowledge to physical setups and carrying out multi-step mathematical reasoning. Each problem includes a worked solution and independently checked reference answers. An LLM-based grading protocol evaluates individual answers and complete multi-part problems, supporting numeric, symbolic, and text responses. We evaluate 53 proprietary and open-weight models. On test problems without figures, high accuracy is concentrated in a small group of thinking models, with the strongest open-weight models approaching proprietary performance. Most open-weight models still solve fewer than half of these problems. Our analysis of Qwen3.5-9B reveals failures in applying physics knowledge and interpreting figures, with misread figures affecting several subquestions.