mlsanityData to 14 Oct 2026

Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models

arXiv 2503.21380 ↗First posted 27 Mar 2025cs.CLcited by 70

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The rapid advancement of large reasoning models has saturated existing math benchmarks, underscoring the urgent need for more challenging evaluation frameworks. To address this, we introduce OlymMATH, a rigorously curated, Olympiad-level math benchmark comprising 350 problems, each with parallel English and Chinese versions. OlymMATH is the first benchmark to unify dual evaluation paradigms within a single suite: (1) natural language evaluation through OlymMATH-EASY and OlymMATH-HARD, comprising 200 computational problems with numerical answers for objective rule-based assessment, and (2) formal verification through OlymMATH-LEAN, offering 150 problems formalized in Lean 4 for rigorous process-level evaluation. All problems are manually sourced from printed publications to minimize data contamination, verified by experts, and span four core domains. Extensive experiments reveal the benchmark's significant challenge, and our analysis also uncovers consistent performance gaps between languages and identifies cases where models employ heuristic "guessing" rather than rigorous reasoning. To further support community research, we release 582k+ reasoning trajectories, a visualization tool, and expert solutions at https://github.com/RUCAIBox/OlymMATH.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 44 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

15 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling19 Dec
  2. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics28 Oct

Other references · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  7. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 192
  8. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  9. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 165
  10. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  11. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  12. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 52
1 more
  1. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 28; 2026: 42.

70 papers.

Sort:

By area:

  1. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  2. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  3. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  4. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  5. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 61
  6. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  7. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 47
  8. 2026MiniMax Sparse Attentioncited by 44
  9. 2025Optimizing Length Compression in Large Reasoning Modelscited by 38
  10. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  11. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematicscited by 31
  12. 2025Nudging the Boundaries of LLM Reasoningcited by 26
38 more
  1. 2025Towards Effective Code-Integrated Reasoningcited by 23
  2. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 19
  3. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMscited by 14
  4. 2025GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMscited by 13
  5. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Modelscited by 13
  6. 2026LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematicscited by 10
  7. 2025Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBenchcited by 8
  8. 2026Riemann-Bench: A Benchmark for Moonshot Mathematicscited by 8
  9. 2026Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoningcited by 8
  10. 2026ATLAS: Verifier-Guided Adaptive Latent Activation Steering for Efficient LLM Reasoningcited by 6
  11. 2026Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspectivecited by 6
  12. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
  13. 2026PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Researchcited by 6
  14. 2026MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrievalcited by 6
  15. 2026OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Modelcited by 5
  16. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Modelcited by 5
  17. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluationcited by 4
  18. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  19. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
  20. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoningcited by 3
  21. 2025WirelessMathLM: Teaching Mathematical Reasoning for LLMs in Wireless Communications with Reinforcement Learningcited by 3
  22. 2025Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoningcited by 3
  23. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversitycited by 3
  24. 2026Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problemscited by 3
  25. 2026ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selectioncited by 3
  26. 2026Unified Data Selection for LLM Reasoningcited by 3
  27. 2025Throttling Web Agents Using Reasoning Gatescited by 2
  28. 2025Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoningcited by 2
  29. 2025Arbitrage: Efficient Reasoning via Advantage-Aware Speculationcited by 2
  30. 2025PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Researchcited by 2
  31. 2026Explicit Multi-head Attention for Inter-head Interaction in Large Language Modelscited by 2
  32. 2026MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubricscited by 2
  33. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  34. 2026TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoningcited by 2
  35. 2026Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarkscited by 2
  36. 2025LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoningcited by 1
  37. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMscited by 1
  38. 2026MathDuels: A Self-Play Benchmark That Growscited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
  2. 2024ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models
  3. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
  4. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  5. 2024FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models
  6. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
  7. 2024Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
  8. 2025EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models