# LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2404.14043 · HTML: https://mlsanity.com/paper/2404.14043/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://mlsanity.com/paper/2305.15294/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework](https://mlsanity.com/paper/2305.03268/) (2023)
- [Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading](https://mlsanity.com/paper/2310.05029/) (2023)
- [Multi-hop Reading Comprehension through Question Decomposition and Rescoring](https://mlsanity.com/paper/1906.02916/) (2019)
- [Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering](https://mlsanity.com/paper/2308.13259/) (2023)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models](https://mlsanity.com/paper/2009.00751/) (2020)
- [SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Bases](https://mlsanity.com/paper/2003.13956/) (2020)

## Built on by (most cited first, 17 of 17)

- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models](https://mlsanity.com/paper/2407.06089/) (2024)
- [Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2411.19443/) (2024)
- [SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation](https://mlsanity.com/paper/2406.19215/) (2024)
- [TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster](https://mlsanity.com/paper/2503.07649/) (2025)
- [Removal of Hallucination on Hallucination: Debate-Augmented RAG](https://mlsanity.com/paper/2505.18581/) (2025)
- [CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Control](https://mlsanity.com/paper/2405.18727/) (2024)
- [Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control](https://mlsanity.com/paper/2502.12145/) (2025)
- [Beyond Prompting: An Efficient Embedding Framework for Open-Domain Question Answering](https://mlsanity.com/paper/2503.01606/) (2025)
- [IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering](https://mlsanity.com/paper/2608.13588/) (2026)
- [ICA-RAG: Information Completeness Guided Adaptive Retrieval-Augmented Generation for Disease Diagnosis](https://mlsanity.com/paper/2502.14614/) (2025)
- [Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates](https://mlsanity.com/paper/2508.01159/) (2025)
- [Aligning Probabilistic Beliefs under Informative Missingness: LLM Steerability in Clinical Reasoning](https://mlsanity.com/paper/2512.00479/) (2025)
- [SERC: LDPC-Inspired Semantic Error Correction for Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.28837/) (2026)
- [PlanE: Meta Planning of Data, Tuning, and Inference for Extractive-based LLMs](https://mlsanity.com/paper/2607.20470/) (2026)
- [EviReform: Evidence-Guided Query Reformulation for Multi-Hop Graph Retrieval](https://mlsanity.com/paper/2608.13006/) (2026)
- [Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoning](https://mlsanity.com/paper/2608.16554/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) demonstrates great value in alleviating outdated knowledge or hallucination by supplying LLMs with updated and relevant knowledge. However, there are still several difficulties for RAG in understanding complex multi-hop query and retrieving relevant documents, which require LLMs to perform reasoning and retrieve step by step. Inspired by human's reasoning process in which they gradually search for the required information, it is natural to ask whether the LLMs could notice the missing information in each reasoning step. In this work, we first experimentally verified the ability of LLMs to extract information as well as to know the missing. Based on the above discovery, we propose a Missing Information Guided Retrieve-Extraction-Solving paradigm (MIGRES), where we leverage the identification of missing information to generate a targeted query that steers the subsequent knowledge retrieval. Besides, we design a sentence-level re-ranking filtering approach to filter the irrelevant content out from document, along with the information extraction capability of LLMs to extract useful information from cleaned-up documents, which in turn to bolster the overall efficacy of RAG. Extensive experiments conducted on multiple public datasets reveal the superiority of the proposed MIGRES method, and analytical experiments demonstrate the effectiveness of our proposed modules.