mlsanityData to 14 Oct 2026

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

arXiv 2606.03858 ↗First posted 2 Jun 2026cs.AIcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks. We introduce PyraMathBench, a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities. Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions. To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection). Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 47 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

25 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data3 Nov
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  3. 2023How well do Large Language Models perform in Arithmetic tasks?16 Mar

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  9. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  10. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  11. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  12. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 165
10 more
  1. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 154
  2. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  3. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  4. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60
  5. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  6. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  7. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42
  8. 2021MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Educationcited by 37
  9. 2019Learning Numeral Embeddingscited by 15
  10. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  2. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  3. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  4. 2024ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models
  5. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  6. 2024Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
  7. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Models
  8. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models