mlsanityData to 7 Oct 2026

ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving

arXiv 2501.08203 ↗First posted 14 Jan 2025cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) have shown impressive capabilities in math problem-solving tasks, their robustness to noisy inputs is not well-studied. We propose ArithmAttack to examine how robust the LLMs are when they encounter noisy prompts that contain extra noise in the form of punctuation marks. While being easy to implement, ArithmAttack does not cause any information loss since words are not added or deleted from the context. We evaluate the robustness of eight LLMs, including LLama3, Mistral, Mathstral, and DeepSeek on noisy GSM8K and MultiArith datasets. Our experiments suggest that all the studied models show vulnerability to such noise, with more noise leading to poorer performances.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 16 of its 25 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

16 of 25 references.

References · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  6. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  7. 2016Solving General Arithmetic Word Problemscited by 353
  8. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  9. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  10. 2017Deceiving Google's Perspective API Built for Detecting Toxic Commentscited by 99
  11. 2021AEDA: An Easier Data Augmentation Technique for Text Classificationcited by 67
  12. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
4 more
  1. 2020From Hero to Z'eroe: A Benchmark of Low-Level Adversarial Attackscited by 36
  2. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31
  3. 2024Reasoning Robustness of LLMs to Adversarial Typographical Errorscited by 31
  4. 2024Give me a hint: Can LLMs take a hint to solve math problems?cited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  2. 2026Evaluating Memory Structure in LLM Agentscited by 12
  3. 2025Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Modelscited by 8
  4. 2025Toward Automated Robustness Evaluation of Mathematical Reasoningcited by 6
  5. 2025Fine-Tuning on Noisy Instructions: Effects on Generalization and Performancecited by 2
  6. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluationcited by 2
  7. 2025Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languagescited by 1
  8. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoningcited by 1
  9. 2025ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluationcited by 1
  10. 2025More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persistscited by 1
  11. 2026Trapped by simplicity: When Transformers fail to learn from noisy featurescited by 0
  12. 2026When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Textcited by 0
1 more
  1. 2026Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMscited by 0