# RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues

arXiv: https://arxiv.org/abs/2409.12558 · HTML: https://mlsanity.com/paper/2409.12558/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [RAFT: Adapting Language Model to Domain Specific RAG](https://mlsanity.com/paper/2403.10131/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [NATURAL PLAN: Benchmarking LLMs on Natural Language Planning](https://mlsanity.com/paper/2406.04520/) (2024)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [Open-Retrieval Conversational Question Answering](https://mlsanity.com/paper/2005.11364/) (2020)
- [Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizations](https://mlsanity.com/paper/2310.05421/) (2023)
- [MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents](https://mlsanity.com/paper/2109.12595/) (2021)
- [From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting](https://mlsanity.com/paper/2309.04269/) (2023)
- [Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning](https://mlsanity.com/paper/2402.14856/) (2024)

## Built on by (most cited first, 19 of 19)

- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge](https://mlsanity.com/paper/2603.04370/) (2026)
- [Benchmarking Poisoning Attacks against Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.18543/) (2025)
- [Vision Language Models in Medicine](https://mlsanity.com/paper/2503.01863/) (2025)
- [MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations](https://mlsanity.com/paper/2602.23184/) (2026)
- [ContextPilot: Fast Long-Context Inference via Context Reuse](https://mlsanity.com/paper/2511.03475/) (2025)
- [Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets](https://mlsanity.com/paper/2504.20119/) (2025)
- [TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing](https://mlsanity.com/paper/2605.18859/) (2026)
- [DRAGOn: Designing RAG On Periodically Updated Corpus](https://mlsanity.com/paper/2507.05713/) (2025)
- [Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks](https://mlsanity.com/paper/2509.22258/) (2025)
- [RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation](https://mlsanity.com/paper/2605.04523/) (2026)
- [A Survey of Query Optimization in Large Language Models](https://mlsanity.com/paper/2412.17558/) (2024)
- [CID-GraphRAG: Enhancing Multi-Turn Dialogue Systems through Dual-Pathway Retrieval of Conversation Flow and Context Semantics](https://mlsanity.com/paper/2506.19385/) (2025)
- [RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits](https://mlsanity.com/paper/2508.19272/) (2025)
- [H-RAG at SemEval-2026 Task 8: Hierarchical Parent-Child Retrieval for Multi-Turn RAG Conversations](https://mlsanity.com/paper/2605.00631/) (2026)
- [A Longitudinal Study on Different Annotator Feedback Loops in Complex RAG Tasks](https://mlsanity.com/paper/2510.11897/) (2025)
- [AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations](https://mlsanity.com/paper/2603.10524/) (2026)
- [uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking](https://mlsanity.com/paper/2606.11945/) (2026)
- [API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces](https://mlsanity.com/paper/2609.08861/) (2026)

## Abstract (arXiv)

In real-world applications with Large Language Models (LLMs), external retrieval mechanisms - such as Search-Augmented Generation (SAG), tool utilization, and Retrieval-Augmented Generation (RAG) - are often employed to enhance the quality of augmented generations in dialogues. These approaches often come with multi-turn dialogue, where each interaction is enriched by relevant information retrieved from external sources. Existing benchmarks either assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in single-turn settings. However, there is a gap in evaluating LLMs' ability to leverage retrieval for more precise responses across multiple turns. To address this limitation, we introduce RAD-Bench (Retrieval Augmented Dialogue), a benchmark designed to evaluate LLMs' capabilities in multi-turn dialogues following retrievals, essential for their deployment in context-rich applications. RAD-Bench evaluates two key abilities of LLMs: Retrieval Synthesis and Retrieval Reasoning. These are measured using discriminative questions and retrieved contexts, and corresponding reference answers, assessing how effectively LLMs integrate and reason with context to maintain and enhance conversation quality over multiple turns. Our evaluation results on commonly used LLMs reveal that model performance deteriorates as additional layers of conditions or constraints are applied across conversation turns, even when accurate retrieved contexts are provided. The data and code are available at https://github.com/mtkresearch/RAD-Bench