# Toward Automated Robustness Evaluation of Mathematical Reasoning

arXiv: https://arxiv.org/abs/2506.05038 · HTML: https://mlsanity.com/paper/2506.05038/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 21 of 21)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations](https://mlsanity.com/paper/2502.06453/) (2025)
- [Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap](https://mlsanity.com/paper/2402.19450/) (2024)
- [Self-Supervised Prompt Optimization](https://mlsanity.com/paper/2502.06855/) (2025)
- [Benchmarking Reasoning Robustness in Large Language Models](https://mlsanity.com/paper/2503.04550/) (2025)
- [GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?](https://mlsanity.com/paper/2502.05252/) (2025)
- [ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving](https://mlsanity.com/paper/2501.08203/) (2025)
- [Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions](https://mlsanity.com/paper/2411.10163/) (2024)
- [MASTER: Enhancing Large Language Model via Multi-Agent Simulated Teaching](https://mlsanity.com/paper/2506.02689/) (2025)

## Built on by (most cited first, 6 of 6)

- [An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems](https://mlsanity.com/paper/2508.08833/) (2025)
- [Robust Reasoning Benchmark](https://mlsanity.com/paper/2604.08571/) (2026)
- [ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation](https://mlsanity.com/paper/2510.08569/) (2025)
- [MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning](https://mlsanity.com/paper/2510.06430/) (2025)
- [RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoning](https://mlsanity.com/paper/2511.04120/) (2025)
- [VeRA: Renewing Reasoning Benchmarks with Executable Specifications](https://mlsanity.com/paper/2602.13217/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning-intensive tasks. However, these models exhibit unexpected brittleness, often failing on simple variations of the same underlying task. Existing robustness evaluations predominantly rely on hand-crafted templates or a limited set of perturbation rules. Consequently, such approaches lack the adaptability to probe latent vulnerabilities unique to specific models and remain susceptible to data contamination. To address this, we propose the Math Stress Tester (MaSTer), an automated framework inspired by software stress testing. MaSTer generates adversarial variants via a multi-round rewrite-verify loop, ensuring semantic consistency while successfully inducing model failure. Our framework generates benchmark variants dynamically for each LLM, thus minimizing the risk of data contamination. Experiments on GSM8K and MATH-500 demonstrate the effectiveness of MaSTer on mathematical tasks. Additionally, we validate the framework's extensibility to non-mathematical tasks, highlighting its broad applicability. Furthermore, we demonstrate that the synthesized variants generated by MaSTer can be utilized as a fine-tuning dataset to significantly enhance the model's robustness.