# CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics

arXiv: https://arxiv.org/abs/2508.18124 · HTML: https://mlsanity.com/paper/2508.18124/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2025.

## Built on by (most cited first, 9 of 9)

- [CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers](https://mlsanity.com/paper/2510.05228/) (2025)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [Group Entropy-Controlled Policy Optimization](https://mlsanity.com/paper/2607.16850/) (2026)
- [PolyReal: A Benchmark for Real-World Polymer Science Workflows](https://mlsanity.com/paper/2604.02934/) (2026)
- [The Agentification of Scientific Research: A Physicist's Perspective](https://mlsanity.com/paper/2604.14718/) (2026)
- [MolViBench: Evaluating LLMs on Molecular Vibe Coding](https://mlsanity.com/paper/2605.02351/) (2026)
- [Rethinking Scientific Discovery in the Agentic Era](https://mlsanity.com/paper/2607.03863/) (2026)
- [Measuring the Partial-Credit Gap: A Strict Benchmark on Vietnam's 2025 Convex Marking Scheme](https://mlsanity.com/paper/2608.18336/) (2026)
- [RSI-Master: Structuring Experiments to Guide Autonomous Model Improvement](https://mlsanity.com/paper/2609.35561/) (2026)

## Abstract (arXiv)

We introduce CMPhysBench, designed to assess the proficiency of Large Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark. CMPhysBench is composed of more than 520 graduate-level meticulously curated questions covering both representative subfields and foundational theoretical frameworks of condensed matter physics, such as magnetism, superconductivity, strongly correlated systems, etc. To ensure a deep understanding of the problem-solving process,we focus exclusively on calculation problems, requiring LLMs to independently generate comprehensive solutions. Meanwhile, leveraging tree-based representations of expressions, we introduce the Scalable Expression Edit Distance (SEED) score, which provides fine-grained (non-binary) partial credit and yields a more accurate assessment of similarity between prediction and ground-truth. Our results show that even the best models, Grok-4, reach only 36 average SEED score and 28% accuracy on CMPhysBench, underscoring a significant capability gap, especially for this practical and frontier domain relative to traditional physics. The code anddataset are publicly available at https://github.com/CMPhysBench/CMPhysBench.