mlsanityData to 14 Oct 2026

How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark

arXiv 2505.18761 ↗First posted 24 May 2025cs.CL cs.AI cs.LGcited by 28

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We introduce Grade School Math with Distracting Context (GSM-DC), a synthetic benchmark to evaluate Large Language Models' (LLMs) reasoning robustness against systematically controlled irrelevant context (IC). GSM-DC constructs symbolic reasoning graphs with precise distractor injections, enabling rigorous, reproducible evaluation. Our experiments demonstrate that LLMs are significantly sensitive to IC, affecting both reasoning path selection and arithmetic accuracy. Additionally, training models with strong distractors improves performance in both in-distribution and out-of-distribution scenarios. We further propose a stepwise tree search guided by a process reward model, which notably enhances robustness in out-of-distribution conditions.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 50 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

29 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?7 Feb
  2. 2024UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts11 Nov
  3. 2024VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning7 Jun
  4. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  7. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  8. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  9. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  10. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  11. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  12. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
13 more
  1. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  2. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  3. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  4. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  5. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  6. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  7. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  8. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  9. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  10. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 36
  11. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  12. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  13. 2024Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Informationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 17.

28 papers.

Sort:

By area:

  1. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  2. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  3. 2026Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agentscited by 11
  4. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 11
  5. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
  6. 2025Shallow Robustness, Deep Vulnerabilities: Multi-Turn Evaluation of Medical LLMscited by 8
  7. 2025Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?cited by 6
  8. 2025DTS: Enhancing Large Reasoning Models via Decoding Tree Sketchingcited by 6
  9. 2025Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoningcited by 5
  10. 2026The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agentscited by 5
  11. 2025A Survey of Link Prediction in N-ary Knowledge Graphscited by 4
  12. 2025Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustnesscited by 4
16 more
  1. 2026Diagnosing and Mitigating Context Rot in Long-horizon Searchcited by 3
  2. 2026Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correctioncited by 3
  3. 2025LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiabilitycited by 2
  4. 2026Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Lengthcited by 2
  5. 2026The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Contextcited by 2
  6. 2025LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuningcited by 1
  7. 2026The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoningcited by 1
  8. 2026Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAGcited by 1
  9. 2026Enhancing Mathematical Problem Solving in LLMs through Execution-Driven Reasoning Augmentationcited by 0
  10. 2026Learning Evidence Highlighting for Frozen LLMscited by 0
  11. 2026Multilingual Reasoning Cascades Need More Contextcited by 0
  12. 2026Implicit Reasoning Steering via Concept Chainingcited by 0
  13. 2026Evaluating Context Segmentation in Locally Deployable SLMs for Cybersecurity CTF Taskscited by 0
  14. 2026Surprising Effectiveness of Self-Demonstrations in Enhancing Schema-Ontology Mapping with LLMscited by 0
  15. 2026AwarenessBench: Assessing Cognitive Capabilities of Language Modelscited by 0
  16. 2026State Representation Shapes Long-Horizon Agent Learning and Transfer: An Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Context
  2. 2026DAGGER: Distractor-Aware Graph Generation for Executable Reasoning in Math Problems
  3. 2026A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoning
  4. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  5. 2026Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation
  6. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  7. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  8. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models