mlsanityData to 7 Oct 2026

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

arXiv 2501.03468 ↗First posted 7 Jan 2025cs.CL cs.AIcited by 50

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 46 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

35 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues19 Sep

Other references · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2024Mixtral of Expertscited by 1,785
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  9. 2018CoQA: A Conversational Question Answering Challengecited by 674
  10. 2019ELI5: Long Form Question Answeringcited by 567
  11. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  12. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
22 more
  1. 2023Active Retrieval Augmented Generationcited by 506
  2. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  3. 2018QuAC : Question Answering in Contextcited by 379
  4. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  5. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  6. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  7. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  8. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  9. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  10. 2020Open-Retrieval Conversational Question Answeringcited by 99
  11. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  12. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  13. 2018Interpretation of Natural Language Rules in Conversational Machine Readingcited by 72
  14. 2021MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documentscited by 53
  15. 2024Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problemcited by 42
  16. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 33
  17. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  18. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 23
  19. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  20. 2024TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistantscited by 16
  21. 2021More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answeringcited by 8
  22. 2024Multi-Document Grounded Multi-Turn Synthetic Dialog Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 27.

50 papers.

Sort:

By area:

  1. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  3. 2025CiteEval: Principle-Driven Citation Evaluation for Source Attributioncited by 19
  4. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  5. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  6. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  7. 2025Activated LoRA: Fine-tuned LLMs for Intrinsicscited by 9
  8. 2025Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggetscited by 8
  9. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  10. 2025Conversational Search: From Fundamentals to Frontiers in the LLM Eracited by 8
  11. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  12. 2026MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversationscited by 8
38 more
  1. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  2. 2025Granite Embedding R2 Modelscited by 7
  3. 2025SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluationcited by 6
  4. 2025DRAGOn: Designing RAG On Periodically Updated Corpuscited by 5
  5. 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routingcited by 4
  6. 2026RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generationcited by 3
  7. 2025FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generationcited by 2
  8. 2025B+ANN: A Fast Billion-Scale Disk-based Nearest-Neighbor Indexcited by 2
  9. 2026Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversationscited by 2
  10. 2025CID-GraphRAG: Enhancing Multi-Turn Dialogue Systems through Dual-Pathway Retrieval of Conversation Flow and Context Semanticscited by 1
  11. 2025RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Editscited by 1
  12. 2026NC-Bench: An LLM Benchmark for Evaluating Conversational Competencecited by 1
  13. 2026DQA: Diagnostic Question Answering for IT Supportcited by 1
  14. 2026H-RAG at SemEval-2026 Task 8: Hierarchical Parent-Child Retrieval for Multi-Turn RAG Conversationscited by 1
  15. 2026Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMscited by 1
  16. 2026Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?cited by 1
  17. 2026K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Financecited by 1
  18. 2026RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLMcited by 1
  19. 2026Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiabilitycited by 1
  20. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  21. 2025A Library of LLM Intrinsics for Retrieval-Augmented Generationcited by 0
  22. 2025CXMArena: Unified Dataset to benchmark performance in realistic CXM Scenarioscited by 0
  23. 2025Latent Principle Discovery for Language Model Self-Improvementcited by 0
  24. 2025A Longitudinal Study on Different Annotator Feedback Loops in Complex RAG Taskscited by 0
  25. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generationcited by 0
  26. 2026AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversationscited by 0
  27. 2026RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generationcited by 0
  28. 2026Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?cited by 0
  29. 2026Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Rerankingcited by 0
  30. 2026Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAGcited by 0
  31. 20265ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Controlcited by 0
  32. 2026Little Brains, Big Feats: Exploring Compact Language Modelscited by 0
  33. 2026CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAGcited by 0
  34. 2026RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Modelscited by 0
  35. 2026EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrievalcited by 0
  36. 2026EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Modelscited by 0
  37. 2026When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agentscited by 0
  38. 2026Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  2. 2026MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
  3. 2026Comprehensive Comparison of RAG Methods Across Multi-Domain Conversational QA
  4. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  5. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  6. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  7. 2024In Defense of RAG in the Era of Long-Context Language Models
  8. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers