# Robust Reasoning Benchmark arXiv: https://arxiv.org/abs/2604.08571 ยท HTML: https://mlsanity.com/paper/2604.08571/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. - Builds on 28 of its 60 references that are in the map. - First posted on arXiv 26 Mar 2026. ## Placed in - LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/) ## Builds on (most cited first, 28 of 28) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024) - [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025) - [Recursive Language Models](https://mlsanity.com/paper/2512.24601/) (2025) - [MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations](https://mlsanity.com/paper/2502.06453/) (2025) - [Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap](https://mlsanity.com/paper/2402.19450/) (2024) - [Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models](https://mlsanity.com/paper/2406.02061/) (2024) - [Dynamic Evaluation of Large Language Models by Meta Probing Agents](https://mlsanity.com/paper/2402.14865/) (2024) - [AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration](https://mlsanity.com/paper/2602.03786/) (2026) - [Cutting Through the Noise: Boosting LLM Performance on Math Word Problems](https://mlsanity.com/paper/2406.15444/) (2024) - [RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models](https://mlsanity.com/paper/2406.11020/) (2024) - [Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?](https://mlsanity.com/paper/2503.18018/) (2025) - [Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs](https://mlsanity.com/paper/2510.10276/) (2025) - [Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information](https://mlsanity.com/paper/2408.10615/) (2024) - [An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems](https://mlsanity.com/paper/2508.08833/) (2025) - [Toward Automated Robustness Evaluation of Mathematical Reasoning](https://mlsanity.com/paper/2506.05038/) (2025) - [Are Reasoning LLMs Robust to Interventions on Their Chain-of-Thought?](https://mlsanity.com/paper/2602.07470/) (2026) - [Quantifying artificial intelligence through algorithmic generalization](https://mlsanity.com/paper/2411.05943/) (2024) - [ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark](https://mlsanity.com/paper/2505.23851/) (2025) - [Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations](https://mlsanity.com/paper/2603.03332/) (2026) - [Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2511.08022/) (2025) - [MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning](https://mlsanity.com/paper/2510.06430/) (2025) - [RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoning](https://mlsanity.com/paper/2511.04120/) (2025) - [More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists](https://mlsanity.com/paper/2511.07112/) (2025) - [MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement](https://mlsanity.com/paper/2511.08055/) (2025) ## Built on by (most cited first, 3 of 3) - [MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning](https://mlsanity.com/paper/2608.03882/) (2026) - [BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning](https://mlsanity.com/paper/2608.07742/) (2026) - [Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations](https://mlsanity.com/paper/2608.22138/) (2026) ## Abstract (arXiv) While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting. We introduce the Robust Reasoning Benchmark (RRB), a pipeline of 13 deterministic textual perturbations applied to AIME 2024 and AIME 2025. Evaluating 8 state-of-the-art models, we find that frontier models are largely resilient, with the notable exception of Claude, which categorically refuses many transformed prompts. Open-weights reasoning models exhibit a range of failure modes under structural noise (cognitive thrashing, tokenization breakdown, and reasoning collapse), with up to 54% average accuracy drops across perturbations and up to 100% on some. We further study one of these failure modes in isolation: attention dilution caused by the model's own chain-of-thought. By tasking models with solving multiple independent mathematical problems sequentially within a single context window, we identify Intra-Query Attention Dilution. Open-weights models ranging from 7B to 120B parameters exhibit accuracy decay on subsequent problems, suggesting that intermediate reasoning steps progressively pollute standard dense attention mechanisms. We argue that in order to achieve reliable reasoning, future architectures need to integrate explicit contextual resets within models' own chain-of-thought, leading to open research questions regarding the optimal granularity of reasoning tasks.