mlsanityData to 7 Oct 2026

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

arXiv 2212.10509 ↗First posted 20 Dec 2022cs.CLcited by 911

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, textitwhat to retrieve depends on textitwhat has already been derived, which in turn may depend on textitwhat was previously retrieved. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning. Code, data, and prompts are available at urlhttps://github.com/stonybrooknlp/ircot

Key facts. Cited by 911 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 124 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

31 of 124 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
19 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  6. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  7. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  8. 2022UL2: Unifying Language Learning Paradigmscited by 292
  9. 2022Teaching Small Language Models to Reasoncited by 279
  10. 2022RealTime QA: What's the Answer Right Now?cited by 239
  11. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  12. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  13. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  14. 2022Recitation-Augmented Language Modelscited by 65
  15. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  16. 2019Answering Complex Open-domain Questions Through Iterative Query Generationcited by 60
  17. 2020IIRC: A Dataset of Incomplete Information Reading Comprehension Questionscited by 52
  18. 2019Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answeringcited by 48
  19. 2019Multi-Hop Paragraph Retrieval for Open-Domain Question Answeringcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 70; 2024: 162; 2025: 290; 2026: 389.

911 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2023Active Retrieval Augmented Generationcited by 506
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
38 more
  1. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  2. 2023Augmented Language Models: a Surveycited by 398
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  4. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  5. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  6. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  7. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  8. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  9. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  10. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  11. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  13. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  14. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  15. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  16. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  17. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  18. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  19. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  20. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  21. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  22. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  23. 2023Text Classification via Large Language Modelscited by 104
  24. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  25. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  26. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  27. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  28. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  29. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  30. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  31. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  32. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  33. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  34. 2025MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilotcited by 83
  35. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  36. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  37. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  38. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75

50 most cited of 911 All 911 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Models
  2. 2025StepChain GraphRAG: Reasoning Over Knowledge Graphs for Multi-Hop Question Answering
  3. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering
  4. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  5. 2024CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering
  6. 2025Reasoning on Multiple Needles In A Haystack
  7. 2024Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs
  8. 2025Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Reading