mlsanityData to 7 Oct 2026

Reasoning Robustness of LLMs to Adversarial Typographical Errors

arXiv 2411.05345 ↗First posted 8 Nov 2024cs.CL cs.AIcited by 31

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning using Chain-of-Thought (CoT) prompting. However, CoT can be biased by users' instruction. In this work, we study the reasoning robustness of LLMs to typographical errors, which can naturally occur in users' queries. We design an Adversarial Typo Attack (textttATA) algorithm that iteratively samples typos for words that are important to the query and selects the edit that is most likely to succeed in attacking. It shows that LLMs are sensitive to minimal adversarial typographical changes. Notably, with 1 character edit, Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K, while with 8 character edits the performance further drops to 19.2%. To extend our evaluation to larger and closed-source LLMs, we develop the textttR2ATA benchmark, which assesses models' underlineReasoning underlineRobustness to underlinetextttATA. It includes adversarial typographical questions derived from three widely used reasoning datasets-GSM8K, BBH, and MMLU-by applying textttATA to open-source LLMs. textttR2ATA demonstrates remarkable transferability and causes notable performance drops across multiple super large and closed-source LLMs.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 33 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

12 of 33 references.

References · 12

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Mistral 7Bcited by 3,065
  5. 2024Mixtral of Expertscited by 1,785
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  9. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  10. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 71
  11. 2018Unrestricted Adversarial Examplescited by 40
  12. 2018Non-Negative Networks Against Adversarial Attackscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 19.

31 papers.

Sort:

By area:

  1. 2025ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solvingcited by 13
  2. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  3. 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errorscited by 9
  4. 2025BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Modelscited by 7
  5. 2025Confidence Elicitation: A New Attack Vector for Large Language Modelscited by 4
  6. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generationcited by 4
  7. 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 4
  8. 2026Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluationcited by 4
  9. 2025Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attackscited by 3
  10. 2026Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliabilitycited by 3
  11. 2025Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyondcited by 2
  12. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scalingcited by 2
19 more
  1. 2025Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queriescited by 1
  2. 2026The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errorscited by 1
  3. 2026Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factorscited by 1
  4. 2026Do LLMs Truly Generalize in the Molecular Domain? A Perturbation-Based Analysiscited by 1
  5. 2026Latent Undertow: How Ordinary Typos Break Probescited by 1
  6. 2025Token Constraint Decoding Improves Robustness on Question Answering for Large Language Modelscited by 0
  7. 2025Uncovering Competency Gaps in Large Language Models and Their Benchmarkscited by 0
  8. 2026Structured Semantic Cloaking for Jailbreak Attacks on Large Language Modelscited by 0
  9. 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasoundcited by 0
  10. 2026When Verification Fails: How Compositionally Infeasible Claims Escape Rejectioncited by 0
  11. 2026ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Drivingcited by 0
  12. 2026Emergent retokenization symmetry in large language models: phenomenology and applicationscited by 0
  13. 2026ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressionscited by 0
  14. 2026At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalizationcited by 0
  15. 2026From Plausible to Actionable: A Position on LLM Self-Explanationscited by 0
  16. 2026Quantifying and Mitigating Korean Jamo-Level Typographical Vulnerabilities in Large Language Modelscited by 0
  17. 2026LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuancescited by 0
  18. 2026When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Textcited by 0
  19. 2026Evaluating the Robustness of Japanese LLMs to IME-Related and Typographical Errorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Selective Adversarial Attacks on LLM Benchmarks
  2. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models
  3. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models
  4. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  5. 2026QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  7. 2026Assessing Adversarial Robustness of Latent Reasoning Models
  8. 2024Vulnerability of LLMs to Vertically Aligned Text Manipulations