mlsanityData to 7 Oct 2026

Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models

arXiv 2410.07985 ↗First posted 10 Oct 2024cs.CLcited by 277

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent advancements in large language models (LLMs) have led to significant breakthroughs in mathematical reasoning capabilities. However, existing benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g., OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels, enabling a holistic assessment of model performance in Olympiad-mathematical reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle with highly challenging Olympiad-level problems, with 60.54% and 52.55% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.

Key facts. Cited by 277 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 24 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

14 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  2. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities13 Jan
  3. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models24 May
  4. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 10

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  6. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  7. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  8. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  9. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 51
  10. 2024Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 151; 2026: 122.

277 papers.

Sort:

By area:

  1. 2025Humanity's Last Examcited by 467
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  4. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  5. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  6. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  7. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  8. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  9. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  10. 2025Phi-4-reasoning Technical Reportcited by 146
  11. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  12. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
38 more
  1. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  2. 2025Process Reward Models That Thinkcited by 118
  3. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 109
  4. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  5. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  6. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  7. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  8. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  9. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  10. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  11. 2025Towards Robust Mathematical Reasoningcited by 67
  12. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  13. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  14. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
  15. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learningcited by 51
  16. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  17. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 48
  18. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 48
  19. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 48
  20. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  21. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  22. 2025FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Modelscited by 45
  23. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  24. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 43
  25. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  26. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  27. 2025Reinforcement Pre-Trainingcited by 40
  28. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  29. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 39
  30. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  31. 2025Understanding Tool-Integrated Reasoningcited by 38
  32. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 36
  33. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  34. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 33
  35. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 32
  36. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32
  37. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 32
  38. 2025The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Modelscited by 31

50 most cited of 277 All 277 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  2. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
  3. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  4. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
  5. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  6. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad
  7. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics
  8. 2025Mathematical Computation and Reasoning Errors by Large Language Models