mlsanityData to 14 Oct 2026

Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis

arXiv 2311.00258 ↗First posted 1 Nov 2023cs.CL cs.LGcited by 6

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent advances in prompt engineering enable large language models (LLMs) to solve multi-hop logical reasoning problems with impressive accuracy. However, there is little existing work investigating the robustness of LLMs with few-shot prompting techniques. Therefore, we introduce a systematic approach to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. We include perturbations at multiple levels of abstractions (e.g. lexical perturbations such as typos, and semantic perturbations such as the inclusion of intermediate reasoning steps in the questions) to conduct behavioral analysis on the LLMs. Throughout our experiments, we find that models are more sensitive to certain perturbations such as replacing words with their synonyms. We also demonstrate that increasing the proportion of perturbed exemplars in the prompts improves the robustness of few-shot prompting methods.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 23 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

20 of 23 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  6. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  10. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  11. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  12. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
5 more
  1. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  2. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  3. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  4. 2018Towards Robust Neural Machine Translationcited by 35
  5. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 1.

6 papers.

  1. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  2. 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Setscited by 6
  3. 2024Is There No Such Thing as a Bad Question? H4R: HalluciBot For Ratiocination, Rewriting, Ranking, and Routingcited by 6
  4. 2024A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understandingcited by 2
  5. 2024Harnessing the Power of Noise: A Survey of Techniques and Applicationscited by 0
  6. 2025Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  2. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
  3. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
  4. 2024Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge
  5. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
  6. 2025Reasoning Capabilities and Invariability of Large Language Models
  7. 2025Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis
  8. 2026Small Language Model Helps Resolve Semantic Ambiguity of LLM Prompt