mlsanityData to 14 Oct 2026

MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data

arXiv 2406.18321 ↗First posted 26 Jun 2024cs.CL cs.AIcited by 52

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have significantly advanced natural language understanding and demonstrated strong problem-solving abilities. Despite these successes, most LLMs still struggle with solving mathematical problems due to the intricate reasoning required. This paper investigates the mathematical problem-solving capabilities of LLMs using the newly developed "MathOdyssey" dataset. The dataset includes diverse mathematical problems at high school and university levels, created by experts from notable institutions to rigorously test LLMs in advanced problem-solving scenarios and cover a wider range of subject areas. By providing the MathOdyssey dataset as a resource to the AI community, we aim to contribute to the understanding and improvement of AI capabilities in complex mathematical problem-solving. We conduct benchmarking on open-source models, such as Llama-3 and DBRX-Instruct, and closed-source models from the GPT series and Gemini models. Our results indicate that while LLMs perform well on routine and moderately difficult tasks, they face significant challenges with Olympiad-level problems and complex university-level questions. Our analysis shows a narrowing performance gap between open-source and closed-source models, yet substantial challenges remain, particularly with the most demanding problems. This study highlights the ongoing need for research to enhance the mathematical reasoning of LLMs. The dataset, results, and code are publicly available.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 17 of its 20 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

17 of 20 references.

References · 17

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  12. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
5 more
  1. 2023Llemma: An Open Language Model For Mathematicscited by 388
  2. 2023Mathematical Capabilities of ChatGPTcited by 268
  3. 2023Challenges and Applications of Large Language Modelscited by 254
  4. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 192
  5. 2023Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problemscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 33; 2026: 10.

52 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  2. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  3. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  4. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  5. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  6. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  7. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  8. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 54
  9. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  10. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 43
  11. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  12. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 27
38 more
  1. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  2. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 24
  3. 2025DeepTheorem: Advancing LLM Reasoning for Theorem Proving Through Natural Language and Reinforcement Learningcited by 24
  4. 2025Beyond Final Answers: Evaluating Large Language Models for Math Tutoringcited by 23
  5. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 20
  6. 2025VisNumBench: Evaluating Number Sense of Multimodal Large Language Modelscited by 14
  7. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  8. 2026LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematicscited by 10
  9. 2026Riemann-Bench: A Benchmark for Moonshot Mathematicscited by 8
  10. 2024MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Treecited by 7
  11. 2025Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Modelscited by 6
  12. 2025MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Taskcited by 5
  13. 2025ASyMOB: Algebraic Symbolic Mathematical Operations Benchmarkcited by 5
  14. 2026Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computingcited by 5
  15. 2025Multi-Step Reasoning in Korean and the Emergent Miragecited by 4
  16. 2025WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communicationscited by 4
  17. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  18. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflowscited by 4
  19. 2025Evolving Excellence: Automated Optimization of LLM-based Agentscited by 4
  20. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoningcited by 3
  21. 2025WirelessMathLM: Teaching Mathematical Reasoning for LLMs in Wireless Communications with Reinforcement Learningcited by 3
  22. 2024RoMath: A Mathematical Reasoning Benchmark in Romaniancited by 2
  23. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuningcited by 2
  24. 2025Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzlescited by 2
  25. 2025Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisonscited by 2
  26. 2026Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarkscited by 2
  27. 2025MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Trainingcited by 1
  28. 2025Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Workcited by 1
  29. 2025MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Modelscited by 1
  30. 2025Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solvingcited by 1
  31. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoningcited by 1
  32. 2026EntroCoT: Enhancing Chain-of-Thought via Adaptive Entropy-Guided Segmentationcited by 1
  33. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMscited by 1
  34. 2026RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Datacited by 1
  35. 2026Position: Deployed Reinforcement Learning should be Continualcited by 1
  36. 2025End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approachcited by 0
  37. 2025Generative AI performance in core undergraduate mathematics: a curriculum-level case studycited by 0
  38. 2025LLM-Driven Rubric-Based Assessment of Algebraic Competence in Multi-Stage Block Coding Tasks with Design and Field Evaluationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EEFSUVA: A New Mathematical Olympiad Benchmark
  2. 2025Evaluation of LLMs for mathematical problem solving
  3. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  4. 2024Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments
  5. 2025Large Language Models and Mathematical Reasoning Failures
  6. 2025Mathematical Computation and Reasoning Errors by Large Language Models
  7. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  8. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models