mlsanityData to 14 Oct 2026

UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts

arXiv 2411.07240 ↗First posted 11 Nov 2024cs.CL cs.AIcited by 8

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problems, existing benchmarks such as GSM8K and MATH present limitations, including narrow problem definitions with specific numbers and reliance on predetermined rules that hinder accurate assessments of reasoning and generality. This paper introduces the UTMath Benchmark, a robust evaluation framework designed to assess LLMs through extensive unit tests, with a focus on both the accuracy and generality of model responses. It comprises 1,053 cutting-edge problems spanning nine mathematical domains, with an average of 68 test cases per problem. UTMath is highly challenging, with the best-performing model, o1-mini, solving only 32.57% of the problems, followed by o1-preview at 27.16%, and GPT-4o at 26.93%. Furthermore, we present the Reasoning-to-Coding of Thoughts (RCoT) approach, which encourages LLMs to engage in explicit reasoning prior to code generation, thereby facilitating the production of more sophisticated solutions and enhancing overall performance and efficiency. Additionally, we also release the UTMath-Train training dataset (more than 70k samples), to support the community in further exploring mathematical reasoning. Our benchmark can be accessed via the following link: https://github.com/UTMathGroup/UTMath

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 48 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

29 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist11 Jul
  2. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  3. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers29 Feb
  4. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges31 Jan
  5. 2023ARB: Advanced Reasoning Benchmark for Large Language Models25 Jul

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
12 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  7. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  9. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 59
  10. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  11. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  12. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 28
  2. 2025OckBench: Measuring the Efficiency of LLM Reasoningcited by 11
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  4. 2026WildSci: Advancing Scientific Reasoning from In-the-Wild Literaturecited by 6
  5. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  6. 2025StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Errorcited by 3
  7. 2025RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoningcited by 1
  8. 2026Think2: Grounded Metacognitive Reasoning in Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
  2. 2026Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
  3. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
  4. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  5. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts
  6. 2026EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discovery
  7. 2024ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models
  8. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI