# More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists

arXiv: https://arxiv.org/abs/2511.07112 · HTML: https://mlsanity.com/paper/2511.07112/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Educational agents and math problem solving, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- LLM agents and tool use / [Educational agents and math problem solving](https://mlsanity.com/agents/educational-agents-and-math-problem-solving/)

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Improving Multi-Agent Debate with Sparse Communication Topology](https://mlsanity.com/paper/2406.11776/) (2024)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation](https://mlsanity.com/paper/2402.10178/) (2024)
- [From Hero to Z'eroe: A Benchmark of Low-Level Adversarial Attacks](https://mlsanity.com/paper/2010.05648/) (2020)
- [Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.06504/) (2023)
- [MALLM: Multi-Agent Large Language Models Framework](https://mlsanity.com/paper/2509.11656/) (2025)
- [ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving](https://mlsanity.com/paper/2501.08203/) (2025)
- [SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science](https://mlsanity.com/paper/2503.23314/) (2025)
- [RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates](https://mlsanity.com/paper/2506.11083/) (2025)
- [Exploring Robustness of Multilingual LLMs on Real-World Noisy Data](https://mlsanity.com/paper/2501.08322/) (2025)
- [Modeling Response Consistency in Multi-Agent LLM Systems: A Comparative Analysis of Shared and Separate Context Approaches](https://mlsanity.com/paper/2504.07303/) (2025)
- [Exploring Robustness of LLMs to Paraphrasing Based on Sociodemographic Factors](https://mlsanity.com/paper/2501.08276/) (2025)

## Built on by (most cited first, 1 of 1)

- [Robust Reasoning Benchmark](https://mlsanity.com/paper/2604.08571/) (2026)

## Abstract (arXiv)

When LLM agents work together, they seem to be more powerful than a single LLM in mathematical question answering. However, are they also more robust to adversarial inputs? We investigate this question using adversarially perturbed math questions. These perturbations include punctuation noise with three intensities (10%, 30%, 50%), plus real-world and human-like typos (WikiTypo, R2ATA). Using a unified sampling-and-voting framework (Agent Forest), we evaluate six open-source models (Qwen3-4B/14B, Llama3.1-8B, Mistral-7B, Gemma3-4B/12B) across four benchmarks (GSM8K, MATH, MMLU-Math, MultiArith), with various numbers of agents n = 1,2,5,10,15,20,25. Our findings show that 1) Noise type matters: punctuation noise harm scales with its severity, and the human typos remain the dominant bottleneck, yielding the largest gaps to Clean accuracy and the highest attack success rate (ASR) even with a large number of agents; 2) Collaboration reliably improves accuracy as the number of agents, n, increases, with the largest gains from n=1 to n=5 and diminishing returns beyond n≈10. However, the adversarial robustness gap persists regardless of the agent count.