# MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit

arXiv: https://arxiv.org/abs/2404.13925 · HTML: https://mlsanity.com/paper/2404.13925/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 10 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 8 of 8)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline](https://mlsanity.com/paper/2401.08190/) (2024)
- [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023)

## Built on by (most cited first, 14 of 14)

- [AlphaMath Almost Zero: Process Supervision without Process](https://mlsanity.com/paper/2405.03553/) (2024)
- [Step-level Value Preference Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2406.10858/) (2024)
- ["Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models](https://mlsanity.com/paper/2406.19417/) (2024)
- [Markov Chain of Thought for Efficient Mathematical Reasoning](https://mlsanity.com/paper/2410.17635/) (2024)
- [ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain](https://mlsanity.com/paper/2411.16736/) (2024)
- [Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2410.10735/) (2024)
- [CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks](https://mlsanity.com/paper/2409.08642/) (2024)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models](https://mlsanity.com/paper/2511.11134/) (2025)
- [MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2406.15468/) (2024)
- [PIPA: Preference Alignment as Prior-Informed Statistical Estimation](https://mlsanity.com/paper/2502.05773/) (2025)
- [Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study](https://mlsanity.com/paper/2506.13464/) (2025)
- [Spurious Advantage Hidden in GRPO](https://mlsanity.com/paper/2609.04063/) (2026)
- [When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Models](https://mlsanity.com/paper/2609.19671/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been explored in a variety of reasoning tasks including solving of mathematical problems. Each math dataset typically includes its own specially designed evaluation script, which, while suitable for its intended use, lacks generalizability across different datasets. Consequently, updates and adaptations to these evaluation tools tend to occur without being systematically reported, leading to inconsistencies and obstacles to fair comparison across studies. To bridge this gap, we introduce a comprehensive mathematical evaluation toolkit that not only utilizes a python computer algebra system (CAS) for its numerical accuracy, but also integrates an optional LLM, known for its considerable natural language processing capabilities. To validate the effectiveness of our toolkit, we manually annotated two distinct datasets. Our experiments demonstrate that the toolkit yields more robust evaluation results compared to prior works, even without an LLM. Furthermore, when an LLM is incorporated, there is a notable enhancement. The code for our method will be made available at urlhttps://github.com/MARIO-Math-Reasoning/mathevaluation.