mlsanityData to 14 Oct 2026

Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

arXiv 2509.26574 ↗First posted 30 Sep 2025cs.AI cond-mat.other cs.CLcited by 21

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 305 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

51 of 305 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models22 Apr
  2. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics19 Feb
  3. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models1 Feb

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2024GPT-4o System Cardcited by 4,588
36 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  11. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  12. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  13. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  14. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  15. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  16. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  17. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  18. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  19. 2025Why Language Models Hallucinatecited by 220
  20. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  21. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  22. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  23. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  24. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  25. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  26. 2025Competitive Programming with Large Reasoning Modelscited by 119
  27. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  28. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  29. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  30. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  31. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  32. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  33. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  34. 2025CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoningcited by 28
  35. 2024The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic reviewcited by 21
  36. 2025Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transitioncited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 19.

21 papers.

Sort:

By area:

  1. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  2. 2026From Procedural Skills to Strategy Genes: Towards Experience-Driven Test-Time Evolutioncited by 9
  3. 2026Hidden Decoding at Scale: Latent Computation Scaling for Large Language Modelscited by 4
  4. 2025QuantumBench: A Benchmark for Quantum Problem Solvingcited by 3
  5. 2026Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraintscited by 3
  6. 2026gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomycited by 3
  7. 2026MaD Physics: Evaluating information seeking under constraints in physical environmentscited by 2
  8. 2026Perfect score on IPhO 2025 theory by Gemini agentcited by 1
  9. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  10. 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMscited by 1
  11. 2026EvalResearchBench: Can AI Agents Design Their Own Evaluations?cited by 1
  12. 2025False Fixed Points: Kantian Feedback, Stable Miscalibration, and Representational Compression in LLMscited by 0
9 more
  1. 2026BioAlchemy: Distilling Biological Literature into Reasoning-Ready Reinforcement Learning Training Datacited by 0
  2. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  3. 2026DualityCert: Verifier-Gated Language-Model Repair of Broken Duality Claims in Quantum Field Theorycited by 0
  4. 2026Motif 3: Technical Reportcited by 0
  5. 2026The Reasoning Tax: Token Economics of LLM Reasoning Across Task Types and Deployment Contextscited by 0
  6. 2026One Capability or Many? Structural and Predictive Tests of Benchmark Validity Disagree About Economic Benchmarks for Frontier AIcited by 0
  7. 2026How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarkscited by 0
  8. 2026Language models can notice an impossible engineering problem yet still report it as solvedcited by 0
  9. 2026The Future of AI-assisted Physics: Discovering, Verifying and Organizing Results via Formalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
  2. 2025Can Theoretical Physics Research Benefit from Language Agents?
  3. 2025Test-time Scaling Techniques in Theoretical Physics -- A Comparison of Methods on the TPBench Dataset
  4. 2025Scaling Physical Reasoning with the PHYSICS Dataset
  5. 2025Symbolic or Numerical? Understanding Physics Problem Solving in Reasoning LLMs
  6. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Tasks
  7. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
  8. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level