mlsanityData to 14 Oct 2026

MathArena: Evaluating LLMs on Uncontaminated Math Competitions

arXiv 2505.23281 ↗First posted 29 May 2025cs.AI cs.CLcited by 297

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The rapid advancement of reasoning capabilities in large language models (LLMs) has led to notable improvements on mathematical benchmarks. However, many of the most commonly used evaluation datasets (e.g., AIME 2024) are widely available online, making it difficult to disentangle genuine reasoning from potential memorization. Furthermore, these benchmarks do not evaluate proof-writing capabilities, which are crucial for many mathematical tasks. To address this, we introduce MathArena, a new benchmark based on the following key insight: recurring math competitions provide a stream of high-quality, challenging problems that can be used for real-time evaluation of LLMs. By evaluating models as soon as new problems are released, we effectively eliminate the risk of contamination. Using this framework, we find strong signs of contamination in AIME 2024. Nonetheless, evaluations on harder competitions, such as CMIMC 2025, demonstrate impressive reasoning capabilities in top-performing models. MathArena is also the first benchmark for proof-writing capabilities. On IMO 2025, top models achieve slightly less than 40%, demonstrating both notable progress and significant room for improvement. So far, we have evaluated over 50 models across seven competitions, totaling 162 problems. As an evolving benchmark, MathArena will continue to track the progress of LLMs on newly released competitions, ensuring rigorous and up-to-date evaluation of mathematical reasoning.

Key facts. Cited by 297 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 35 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

20 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics1 Apr
  2. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad27 Mar
  3. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models27 Mar
  4. 2025MathConstruct: Challenging LLM Reasoning with Constructive Proofs14 Feb
  5. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations10 Feb
  6. 2025Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation24 Jan
  7. 2024HARP: A challenging human-annotated math reasoning benchmark11 Dec
  8. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI7 Nov
  9. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models10 Oct
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
  11. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic1 May

Other references · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  5. 2023Mathematical Capabilities of ChatGPTcited by 268
  6. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 165
  7. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  8. 2024Lean Workbook: A large-scale Lean problem set formalized from natural language math problemscited by 122
  9. 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 66; 2026: 231.

297 papers.

Sort:

By area:

  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  2. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
  3. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  4. 2025Deep Think with Confidencecited by 205
  5. 2026MiMo-V2-Flash Technical Reportcited by 191
  6. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  7. 2026Qwen3.5-Omni Technical Reportcited by 174
  8. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  9. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  10. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  11. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  12. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
38 more
  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  2. 2026Qwen3-Coder-Next Technical Reportcited by 71
  3. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  4. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
  5. 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 60
  6. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  7. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  8. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  9. 2025The Majority is not always right: RL training for solution aggregationcited by 47
  10. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  11. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  12. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  13. 2026Maximum Likelihood Reinforcement Learningcited by 36
  14. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMscited by 35
  15. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  16. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  17. 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 30
  18. 2025Single-stream Policy Optimizationcited by 28
  19. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 27
  20. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 26
  21. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  22. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 24
  23. 2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokenscited by 22
  24. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  25. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  26. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  27. 2025Reliable Fine-Grained Evaluation of Natural Language Math Proofscited by 18
  28. 2026MEMENTO: Teaching LLMs to Manage Their Own Contextcited by 18
  29. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 18
  30. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 16
  31. 2025Lost at the Beginning of Reasoningcited by 15
  32. 2025OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Valuecited by 15
  33. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 15
  34. 2025Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuitscited by 14
  35. 2026Parallel-Probe: Towards Efficient Parallel Thinking via 2D Probingcited by 14
  36. 2026LLMs Encode Their Failures: Predicting Success from Pre-Generation Activationscited by 14
  37. 2025IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generationcited by 13
  38. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
  2. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  3. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  4. 2025Can LLMs textitunderstand Math? -- Exploring the Pitfalls in Mathematical Reasoning
  5. 2025Evaluating the Reasoning Abilities of LLMs on Underrepresented Mathematics Competition Problems
  6. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics
  7. 2024Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
  8. 2025SKYLENAGE Technical Report: Mathematical Reasoning and Contest-Innovation Benchmarks for Multi-Level Math Evaluation