mlsanityData to 7 Oct 2026

FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

arXiv 2411.04872 ↗First posted 7 Nov 2024cs.AIcited by 227

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.

Key facts. Cited by 227 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 31 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

14 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models10 Oct
  2. 2023ARB: Advanced Reasoning Benchmark for Large Language Models25 Jul

Other references · 12

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  6. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  7. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  8. 2023Mathematical Capabilities of ChatGPTcited by 263
  9. 2024Are We Done with MMLU?cited by 178
  10. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 153
  11. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  12. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 115; 2026: 108.

227 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2025Humanity's Last Examcited by 467
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  5. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  8. 2025BIG-Bench Extra Hardcited by 123
  9. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  10. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  11. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  12. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
38 more
  1. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  2. 2025Towards Robust Mathematical Reasoningcited by 67
  3. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  4. 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 55
  5. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  6. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  7. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  8. 2025Reasoning Language Models: A Blueprintcited by 41
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  11. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 36
  12. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  13. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  14. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 32
  15. 2025SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domaincited by 30
  16. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  17. 2025A Rosetta Stone for AI Benchmarkscited by 29
  18. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  19. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 27
  20. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  21. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 25
  22. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  23. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  24. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 23
  25. 2025FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levelscited by 23
  26. 2025Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapointscited by 22
  27. 2025Sudoku-Bench: Evaluating creative reasoning with Sudoku variantscited by 21
  28. 2026Automated Conjecture Resolution with Formal Verificationcited by 21
  29. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 19
  30. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 19
  31. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 18
  32. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  33. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  34. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 16
  35. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  36. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Sciencecited by 15
  37. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 14
  38. 2026AI co-mathematician: Accelerating mathematicians with agentic AIcited by 13

50 most cited of 227 All 227 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discovery
  2. 2025Evaluating Frontier LLMs on PhD-Level Mathematical Reasoning: A Benchmark on a Textbook in Theoretical Computer Science about Randomized Algorithms
  3. 2025AI Mathematician: Towards Fully Automated Frontier Mathematical Research
  4. 2024UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts
  5. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
  6. 2025FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
  7. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics
  8. 2024Mathematics and Machine Creativity: A Survey on Bridging Mathematics with AI