# RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

arXiv: https://arxiv.org/abs/2505.12575 · HTML: https://mlsanity.com/paper/2505.12575/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023)
- [MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics](https://mlsanity.com/paper/2109.00110/) (2021)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Learning to Prove Theorems via Interacting with Proof Assistants](https://mlsanity.com/paper/1905.09381/) (2019)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad](https://mlsanity.com/paper/2503.21934/) (2025)
- [Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models](https://mlsanity.com/paper/2503.21380/) (2025)
- [BaxBench: Can LLMs Generate Correct and Secure Backends?](https://mlsanity.com/paper/2502.11844/) (2025)
- [Competition-Level Problems are Effective LLM Evaluators](https://mlsanity.com/paper/2312.02143/) (2023)
- [MathConstruct: Challenging LLM Reasoning with Constructive Proofs](https://mlsanity.com/paper/2502.10197/) (2025)

## Built on by (most cited first, 30 of 31)

- [AMO-Bench: Large Language Models Still Struggle in High School Math Competitions](https://mlsanity.com/paper/2510.26768/) (2025)
- [First Proof](https://mlsanity.com/paper/2602.05192/) (2026)
- [FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels](https://mlsanity.com/paper/2511.02872/) (2025)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Models](https://mlsanity.com/paper/2506.17114/) (2025)
- [Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense](https://mlsanity.com/paper/2510.07242/) (2025)
- [IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation](https://mlsanity.com/paper/2509.26076/) (2025)
- [LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics](https://mlsanity.com/paper/2602.24173/) (2026)
- [ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?](https://mlsanity.com/paper/2608.03874/) (2026)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation](https://mlsanity.com/paper/2512.21002/) (2025)
- [LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches](https://mlsanity.com/paper/2604.01754/) (2026)
- [Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning](https://mlsanity.com/paper/2601.00791/) (2026)
- [The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learning](https://mlsanity.com/paper/2603.15914/) (2026)
- [Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs](https://mlsanity.com/paper/2605.09063/) (2026)
- [Albilich: Steerable Proof-State Orchestration for LLM-Based Mathematical Research with CAS Integration](https://mlsanity.com/paper/2607.27705/) (2026)
- [Test of Time: Rethinking Temporal Signal of Benchmark Contamination](https://mlsanity.com/paper/2509.00072/) (2025)
- [Performance of AI agents based on reasoning language models on ALD process optimization tasks](https://mlsanity.com/paper/2601.09980/) (2026)
- [Reasoning over mathematical objects: on-policy reward modeling and test time aggregation](https://mlsanity.com/paper/2603.18886/) (2026)
- [MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs](https://mlsanity.com/paper/2605.10064/) (2026)
- [ResearchMath-14K: Scaling Research-Level Mathematics via Agents](https://mlsanity.com/paper/2605.28003/) (2026)
- [Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math](https://mlsanity.com/paper/2602.06291/) (2026)
- [MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts](https://mlsanity.com/paper/2604.06505/) (2026)
- [Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning](https://mlsanity.com/paper/2605.09292/) (2026)
- [Benchmarks in Leipzig](https://mlsanity.com/paper/2606.05818/) (2026)
- [CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions](https://mlsanity.com/paper/2606.06526/) (2026)
- [Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs](https://mlsanity.com/paper/2606.15258/) (2026)
- [Failure Modes of Large Language Models on Research-Level Mathematics: A Taxonomy and an Empirical Characterisation](https://mlsanity.com/paper/2606.24902/) (2026)
- [Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics](https://mlsanity.com/paper/2607.06820/) (2026)
- [The Ramanujan Challenge For AI](https://mlsanity.com/paper/2607.09721/) (2026)

## Abstract (arXiv)

Existing benchmarks for evaluating mathematical reasoning in large language models (LLMs) rely primarily on competition problems, formal proofs, or artificially challenging questions -- failing to capture the nature of mathematics encountered in actual research environments. We introduce RealMath, a novel benchmark derived directly from research papers and mathematical forums that assesses LLMs' abilities on authentic mathematical tasks. Our approach addresses three critical challenges: sourcing diverse research-level content, enabling reliable automated evaluation through verifiable statements, and designing a continually refreshable dataset to mitigate contamination risks. Experimental results across multiple LLMs reveal surprising capabilities in handling research mathematics compared to competition problems, suggesting current models may already serve as valuable assistants for working mathematicians despite limitations on highly challenging problems. The code and dataset for RealMath are publicly available.