# Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

arXiv: https://arxiv.org/abs/2409.14924 · HTML: https://mlsanity.com/paper/2409.14924/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 165 of its 192 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 30 of 165)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)

## Built on by (most cited first, 30 of 66)

- [Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs](https://mlsanity.com/paper/2507.09477/) (2025)
- [RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2504.18041/) (2025)
- [Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging](https://mlsanity.com/paper/2505.09316/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey](https://mlsanity.com/paper/2504.10499/) (2025)
- [Context Engineering 2.0: The Context of Context Engineering](https://mlsanity.com/paper/2510.26493/) (2025)
- [O1 Embedder: Let Retrievers Think Before Action](https://mlsanity.com/paper/2502.07555/) (2025)
- [EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora](https://mlsanity.com/paper/2506.20963/) (2025)
- [Credible Plan-Driven RAG Method for Multi-Hop Question Answering](https://mlsanity.com/paper/2504.16787/) (2025)
- [EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving](https://mlsanity.com/paper/2506.02672/) (2025)
- [Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool use](https://mlsanity.com/paper/2602.23368/) (2025)
- [DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent](https://mlsanity.com/paper/2603.01152/) (2026)
- [SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge Refinement](https://mlsanity.com/paper/2504.03561/) (2025)
- [Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding](https://mlsanity.com/paper/2510.15253/) (2025)
- [PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimation](https://mlsanity.com/paper/2504.01509/) (2025)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [QAgent: A modular Search Agent with Interactive Query Understanding](https://mlsanity.com/paper/2510.08383/) (2025)
- [Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2411.19463/) (2024)
- [A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking](https://mlsanity.com/paper/2505.02171/) (2025)
- [Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval](https://mlsanity.com/paper/2505.15753/) (2025)
- [CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning](https://mlsanity.com/paper/2601.14952/) (2026)
- [Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach](https://mlsanity.com/paper/2502.14100/) (2025)
- [Method-Based Reasoning for Large Language Models: Extraction, Reuse, and Continuous Improvement](https://mlsanity.com/paper/2508.04289/) (2025)
- [PG-Agent: An Agent Powered by Page Graph](https://mlsanity.com/paper/2509.03536/) (2025)
- [AVA: Towards Agentic Video Analytics with Vision Language Models](https://mlsanity.com/paper/2505.00254/) (2025)
- [DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization](https://mlsanity.com/paper/2505.10989/) (2025)
- [Enterprise Large Language Model Evaluation Benchmark](https://mlsanity.com/paper/2506.20274/) (2025)
- [Accelerating Prefilling via Decoding-time Contribution Sparsity](https://mlsanity.com/paper/2507.21526/) (2025)
- [MetaAgent: Toward Self-Evolving Agent via Tool Meta-Learning](https://mlsanity.com/paper/2508.00271/) (2025)
- [SafeRBench: Dissecting the Reasoning Safety of Large Language Models](https://mlsanity.com/paper/2511.15169/) (2025)

## Abstract (arXiv)

Large language models (LLMs) augmented with external data have demonstrated remarkable capabilities in completing real-world tasks. Techniques for integrating external data into LLMs, such as Retrieval-Augmented Generation (RAG) and fine-tuning, are gaining increasing attention and widespread application. Nonetheless, the effective deployment of data-augmented LLMs across various specialized fields presents substantial challenges. These challenges encompass a wide range of issues, from retrieving relevant data and accurately interpreting user intent to fully harnessing the reasoning capabilities of LLMs for complex tasks. We believe that there is no one-size-fits-all solution for data-augmented LLM applications. In practice, underperformance often arises from a failure to correctly identify the core focus of a task or because the task inherently requires a blend of multiple capabilities that must be disentangled for better resolution. In this survey, we propose a RAG task categorization method, classifying user queries into four levels based on the type of external data required and primary focus of the task: explicit fact queries, implicit fact queries, interpretable rationale queries, and hidden rationale queries. We define these levels of queries, provide relevant datasets, and summarize the key challenges and most effective techniques for addressing these challenges. Finally, we discuss three main forms of integrating external data into LLMs: context, small model, and fine-tuning, highlighting their respective strengths, limitations, and the types of problems they are suited to solve. This work aims to help readers thoroughly understand and decompose the data requirements and key bottlenecks in building LLM applications, offering solutions to the different challenges and serving as a guide to systematically developing such applications.