# Adaptive Retrieval-Augmented Generation for Conversational Systems

arXiv: https://arxiv.org/abs/2407.21712 · HTML: https://mlsanity.com/paper/2407.21712/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://mlsanity.com/paper/2010.08824/) (2020)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)
- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.13781/) (2024)
- [Learning to Select Knowledge for Response Generation in Dialog Systems](https://mlsanity.com/paper/1902.04911/) (2019)
- [In conversation with Artificial Intelligence: aligning language models with human values](https://mlsanity.com/paper/2209.00731/) (2022)
- [Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access](https://mlsanity.com/paper/2006.03533/) (2020)
- [Entity Cloze By Date: What LMs Know About Unseen Entities](https://mlsanity.com/paper/2205.02832/) (2022)
- [Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2305.18846/) (2023)
- [UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systems](https://mlsanity.com/paper/2401.13256/) (2024)
- [Towards Universal Dialogue State Tracking](https://mlsanity.com/paper/1810.09587/) (2018)
- [Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation](https://mlsanity.com/paper/2112.07924/) (2021)
- [GraphDialog: Integrating Graph Knowledge into End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2010.01447/) (2020)
- [Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check](https://mlsanity.com/paper/2403.18243/) (2024)
- [Language Models in Dialogue: Conversational Maxims for Human-AI Interactions](https://mlsanity.com/paper/2403.15115/) (2024)
- [Towards Zero and Few-shot Knowledge-seeking Turn Detection in Task-orientated Dialogue Systems](https://mlsanity.com/paper/2109.08820/) (2021)
- [Task2KB: A Public Task-Oriented Knowledge Base](https://mlsanity.com/paper/2302.00654/) (2023)

## Built on by (most cited first, 22 of 22)

- [Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges](https://mlsanity.com/paper/2506.10408/) (2025)
- [Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey](https://mlsanity.com/paper/2504.10499/) (2025)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.16415/) (2025)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [MedKGI: Iterative Differential Diagnosis with Medical Knowledge Graphs and Information-Guided Inquiring](https://mlsanity.com/paper/2512.24181/) (2025)
- [Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control](https://mlsanity.com/paper/2502.12145/) (2025)
- [A2RAG: Adaptive Agentic Graph Retrieval for Cost-Aware and Reliable Reasoning](https://mlsanity.com/paper/2601.21162/) (2026)
- [TOBUGraph: Knowledge Graph-Based Retrieval for Enhanced LLM Performance Beyond RAG](https://mlsanity.com/paper/2412.05447/) (2024)
- [Information Retrieval in the Age of Generative AI: The RGB Model](https://mlsanity.com/paper/2504.20610/) (2025)
- [Fast, Slow, and Tool-augmented Thinking for LLMs: A Review](https://mlsanity.com/paper/2508.12265/) (2025)
- [Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities](https://mlsanity.com/paper/2508.20324/) (2025)
- [Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.14546/) (2026)
- [VikingMem: A Memory Base Management System for Stateful LLM-based Applications](https://mlsanity.com/paper/2605.29640/) (2026)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)
- [Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering](https://mlsanity.com/paper/2604.07985/) (2026)
- [Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval](https://mlsanity.com/paper/2510.02938/) (2025)
- [Training the Knowledge Base through Evidence Distillation and Write-Back Enrichment](https://mlsanity.com/paper/2603.25737/) (2026)
- [Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time](https://mlsanity.com/paper/2606.01923/) (2026)
- [Overcoming the Impedance Mismatch: A Theoretical Roadmap for Fusing Foundation Models and Knowledge Graphs](https://mlsanity.com/paper/2606.15656/) (2026)
- [CoBRA: Learning Tool-Use Boundaries via Counterfactual Margins](https://mlsanity.com/paper/2609.00967/) (2026)
- [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026)

## Abstract (arXiv)

Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the always need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop RAGate, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying RAGate to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of RAGate in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses and a high generation confidence. This study also identifies the correlation between the generation's confidence level and the relevance of the augmented knowledge.