mlsanityData to 7 Oct 2026

Large Language Models Can Be Easily Distracted by Irrelevant Context

arXiv 2302.00093 ↗First posted 31 Jan 2023cs.CL cs.AIcited by 820

Not on the map. Why?

Cited by 165 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information.

Key facts. Cited by 820 papers in the map (data through 7 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

49 of 63 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
37 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  10. 2022PAL: Program-aided Language Modelscited by 635
  11. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  12. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  13. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  14. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  15. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  16. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  17. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  18. 2020Transformers as Soft Reasoners over Languagecited by 265
  19. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  20. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  21. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  22. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  23. 2022Large Language Models with Controllable Working Memorycited by 169
  24. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  25. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  26. 2022Natural Language to Code Translation with Executioncited by 122
  27. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  28. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  29. 2022Language Model Cascadescited by 104
  30. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  31. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  32. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  33. 2021Sorting through the noise: Testing robustness of information processing in pre-trained language modelscited by 27
  34. 2021Adversarial Examples for Evaluating Math Word Problem Solverscited by 20
  35. 2022COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Modelscited by 15
  36. 2018Learning Visually-Grounded Semantics from Contrastive Adversarial Samplescited by 13
  37. 2018Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answeringcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 115; 2024: 239; 2025: 207; 2026: 259.

820 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  8. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  9. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  12. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
38 more
  1. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  3. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  4. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  5. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  10. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  11. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  12. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  13. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  14. 2024Corrective Retrieval Augmented Generationcited by 187
  15. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  16. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  17. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  18. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  19. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  20. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  21. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  22. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  23. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  24. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  25. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  26. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  27. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  28. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  29. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  30. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  31. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 121
  32. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  33. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 116
  34. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  35. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  36. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 102
  37. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  38. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100

50 most cited of 820 All 820 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
  2. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  3. 2024Instructing Large Language Models to Identify and Ignore Irrelevant Conditions
  4. 2025Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them
  5. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  6. 2026A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning
  7. 2025LLMs can be easily Confused by Instructional Distractions
  8. 2024Large Language Models Struggle with Unreasonability in Math Problems