# CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers

arXiv: https://arxiv.org/abs/2510.05228 · HTML: https://mlsanity.com/paper/2510.05228/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 17 of 17)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark](https://mlsanity.com/paper/2405.12209/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models](https://mlsanity.com/paper/2504.16074/) (2025)
- [Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics](https://mlsanity.com/paper/2502.15815/) (2025)
- [HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics](https://mlsanity.com/paper/2410.09988/) (2024)
- [Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving](https://mlsanity.com/paper/2503.21821/) (2025)
- [CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning](https://mlsanity.com/paper/2503.13517/) (2025)
- [Quantum Many-Body Physics Calculations with Large Language Models](https://mlsanity.com/paper/2403.03154/) (2024)
- [CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics](https://mlsanity.com/paper/2508.18124/) (2025)
- [PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models](https://mlsanity.com/paper/2505.24823/) (2025)

## Built on by (most cited first, 4 of 4)

- [QMBench: A Research Level Benchmark for Quantum Materials Research](https://mlsanity.com/paper/2512.19753/) (2025)
- [The Agentification of Scientific Research: A Physicist's Perspective](https://mlsanity.com/paper/2604.14718/) (2026)
- [Fine-Tuning Small Reasoning Models for Quantum Field Theory](https://mlsanity.com/paper/2604.18936/) (2026)
- [QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks](https://mlsanity.com/paper/2609.35581/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable progress in coding and math problem-solving, but evaluation on advanced research-level problems in hard sciences remains scarce. To fill this gap, we present CMT-Benchmark, a dataset of 50 problems covering condensed matter theory (CMT) at the level of an expert researcher. Topics span analytical and computational approaches in quantum many-body, and classical statistical mechanics. The dataset was designed and verified by a panel of expert researchers from around the world. We built the dataset through a collaborative environment that challenges the panel to write and refine problems they would want a research assistant to solve, including Hartree-Fock, exact diagonalization, quantum/variational Monte Carlo, density matrix renormalization group (DMRG), quantum/classical statistical mechanics, and model building. We evaluate LLMs by programmatically checking solutions against expert-supplied ground truth. We developed machine-grading, including symbolic handling of non-commuting operators via normal ordering. They generalize across tasks too. Our evaluations show that frontier models struggle with all of the problems in the dataset, highlighting a gap in the physical reasoning skills of current LLMs. Notably, experts identified strategies for creating increasingly difficult problems by interacting with the LLMs and exploiting common failure modes. The best model, GPT5, solves 30% of the problems; average across 17 models (GPT, Gemini, Claude, DeepSeek, Llama) is 11.4±2.1%. Moreover, 18 problems are solved by none of the 17 models, and 26 by at most one. These unsolved problems span Quantum Monte Carlo, Variational Monte Carlo, and DMRG. Answers sometimes violate fundamental symmetries or have unphysical scaling dimensions. We believe this benchmark will guide development toward capable AI research assistants and tutors.