mlsanityData to 14 Oct 2026

Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics

arXiv 2502.15815 ↗First posted 19 Feb 2025cs.LG astro-ph.CO cs.AIcited by 34

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We introduce a benchmark to evaluate the capability of AI to solve problems in theoretical physics, focusing on high-energy theory and cosmology. The first iteration of our benchmark consists of 57 problems of varying difficulty, from undergraduate to research level. These problems are novel in the sense that they do not come from public problem collections. We evaluate our data set on various open and closed language models, including o3-mini, o1, DeepSeek-R1, GPT-4o and versions of Llama and Qwen. While we find impressive progress in model performance with the most recent models, our research-level difficulty problems are mostly unsolved. We address challenges of auto-verifiability and grading, and discuss common failure modes. While currently state-of-the art models are still of limited use for researchers, our results show that AI assisted theoretical physics research may become possible in the near future. We discuss the main obstacles towards this goal and possible strategies to overcome them. The public problems and solutions, results for various models, and updates to the data set and score distribution, are available on the website of the dataset tpbench.org.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 82 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

55 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents1 Dec

Other references · 54

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
38 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  13. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  14. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  15. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  16. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  17. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  18. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  19. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  20. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  21. 2023Do Large Language Models Know What They Don't Know?cited by 243
  22. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  23. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  24. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  25. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 192
  26. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 165
  27. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 159
  28. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  29. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  30. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  31. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  32. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 90
  33. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 77
  34. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 76
  35. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 68
  36. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 60
  37. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  38. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Timecited by 41

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 15.

34 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  4. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 34
  5. 2025FEABench: Evaluating Language Models on Multiphysics Reasoning Abilitycited by 22
  6. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  7. 2026Large Language Model Reasoning Failurescited by 21
  8. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 18
  9. 2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9
  10. 2025Test-time Scaling Techniques in Theoretical Physics -- A Comparison of Methods on the TPBench Datasetcited by 8
  11. 2025QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistrycited by 8
  12. 2025PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problemscited by 7
22 more
  1. 2026Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproductioncited by 7
  2. 2026MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Modelscited by 6
  3. 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Researchcited by 6
  4. 2025HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Classcited by 5
  5. 2026Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computingcited by 5
  6. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Modelscited by 4
  7. 2025CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researcherscited by 4
  8. 2025FeynTune: Large Language Models for High-Energy Theorycited by 3
  9. 2025QMBench: A Research Level Benchmark for Quantum Materials Researchcited by 3
  10. 2026gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomycited by 3
  11. 2025PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Researchcited by 2
  12. 2026LLMs with in-context learning for Algorithmic Theoretical Physicscited by 2
  13. 2025Towards a Large Physics Benchmarkcited by 1
  14. 2025Mastering Olympiad-Level Physics with Artificial Intelligencecited by 1
  15. 2026Bethe Ansatz with a Large Language Modelcited by 1
  16. 2026Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMscited by 1
  17. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  18. 2026AxQM: A Textbook-Scale Benchmark for Formal Proof Synthesis in a Library of Finite-Dimensional Quantum Mechanicscited by 1
  19. 2026When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoningcited by 0
  20. 2026Assessing AI in Introductory Physics Problem Solvingcited by 0
  21. 2026DualityCert: Verifier-Gated Language-Model Repair of Broken Duality Claims in Quantum Field Theorycited by 0
  22. 2026The Future of AI-assisted Physics: Discovering, Verifying and Organizing Results via Formalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving
  2. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level
  3. 2026TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving
  4. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models
  5. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
  6. 2025SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code
  7. 2025Scaling Physical Reasoning with the PHYSICS Dataset
  8. 2026How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks