mlsanityData to 14 Oct 2026

RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

arXiv 2505.12575 ↗First posted 18 May 2025cs.AIcited by 31

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Existing benchmarks for evaluating mathematical reasoning in large language models (LLMs) rely primarily on competition problems, formal proofs, or artificially challenging questions -- failing to capture the nature of mathematics encountered in actual research environments. We introduce RealMath, a novel benchmark derived directly from research papers and mathematical forums that assesses LLMs' abilities on authentic mathematical tasks. Our approach addresses three critical challenges: sourcing diverse research-level content, enabling reliable automated evaluation through verifiable statements, and designing a continually refreshable dataset to mitigate contamination risks. Experimental results across multiple LLMs reveal surprising capabilities in handling research mathematics compared to competition problems, suggesting current models may already serve as valuable assistants for working mathematicians despite limitations on highly challenging problems. The code and dataset for RealMath are publicly available.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 31 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

15 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad27 Mar
  2. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models27 Mar
  3. 2025MathConstruct: Challenging LLM Reasoning with Constructive Proofs14 Feb
  4. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models10 Oct

Other references · 11

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  6. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  7. 2023Mathematical Capabilities of ChatGPTcited by 268
  8. 2019Learning to Prove Theorems via Interacting with Proof Assistantscited by 109
  9. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
  10. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 41
  11. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 22.

31 papers.

Sort:

By area:

  1. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  2. 2026First Proofcited by 32
  3. 2025FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levelscited by 27
  4. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  5. 2025Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Modelscited by 20
  6. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 20
  7. 2025IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generationcited by 13
  8. 2026LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematicscited by 10
  9. 2026ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?cited by 8
  10. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 6
  11. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncationcited by 5
  12. 2026LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketchescited by 5
19 more
  1. 2026Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoningcited by 4
  2. 2026The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learningcited by 4
  3. 2026Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMscited by 4
  4. 2026Albilich: Steerable Proof-State Orchestration for LLM-Based Mathematical Research with CAS Integrationcited by 4
  5. 2025Test of Time: Rethinking Temporal Signal of Benchmark Contaminationcited by 3
  6. 2026Performance of AI agents based on reasoning language models on ALD process optimization taskscited by 3
  7. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  8. 2026MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphscited by 2
  9. 2026ResearchMath-14K: Scaling Research-Level Mathematics via Agentscited by 2
  10. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  11. 2026MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstractscited by 1
  12. 2026Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoningcited by 1
  13. 2026Benchmarks in Leipzigcited by 0
  14. 2026CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussionscited by 0
  15. 2026Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofscited by 0
  16. 2026Failure Modes of Large Language Models on Research-Level Mathematics: A Taxonomy and an Empirical Characterisationcited by 0
  17. 2026Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematicscited by 0
  18. 2026The Ramanujan Challenge For AIcited by 0
  19. 2026SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Outputcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
  2. 2026EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discovery
  3. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models
  4. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning
  5. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  6. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  7. 2024LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
  8. 2026AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification