mlsanityData to 7 Oct 2026

RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems

arXiv 2407.11005 ↗First posted 25 Jun 2024cs.CL cs.AIcited by 115

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 43 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

35 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models17 Apr
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 32

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  11. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  12. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
20 more
  1. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  2. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  3. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  5. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  6. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  8. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  9. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  10. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  11. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  12. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  13. 2023Human Feedback is not Gold Standardcited by 82
  14. 2021Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Featurescited by 74
  15. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  16. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answeringcited by 43
  17. 2019The TechQA Datasetcited by 29
  18. 2023The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answeringcited by 26
  19. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  20. 2021Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Frameworkcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 48; 2026: 61.

115 papers.

Sort:

By area:

  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  2. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  3. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  4. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  5. 2024Is Semantic Chunking Worth the Computational Cost?cited by 43
  6. 2025LLMs for Explainable AI: A Comprehensive Surveycited by 42
  7. 2025GFM-RAG: Graph Foundation Model for Retrieval Augmented Generationcited by 40
  8. 2025CARROT: A Cost Aware Rate Optimal Routercited by 34
  9. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 27
  10. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 24
  11. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
  12. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
38 more
  1. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  3. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
  4. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  5. 2026R2-Router: A New Paradigm for LLM Routing with Reasoningcited by 13
  6. 2025CellForge: Agentic Design of Virtual Cell Modelscited by 11
  7. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 11
  8. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  9. 2025HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunkingcited by 9
  10. 2024Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Studycited by 8
  11. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  12. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  13. 2026HALT: Hallucination Assessment via Log-probs as Time seriescited by 8
  14. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  15. 2025RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systemscited by 7
  16. 2026SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directionscited by 7
  17. 2025HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Taskscited by 6
  18. 2025FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Modelscited by 6
  19. 2026Corpus2Skill: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAGcited by 6
  20. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphscited by 5
  21. 2025Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generationcited by 5
  22. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  23. 2026FinTradeBench: A Financial Reasoning Benchmark for LLMscited by 5
  24. 2025DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimizationcited by 4
  25. 2025RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelinescited by 4
  26. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  27. 2026RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routingcited by 4
  28. 2026Visual Inception: Compromising Long-term Planning in Agentic Recommenders via Multimodal Memory Poisoningcited by 4
  29. 2024How Does Knowledge Selection Help Retrieval Augmented Generation?cited by 3
  30. 2025CLEAR: Error Analysis via LLM-as-a-Judge Made Easycited by 3
  31. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Modelscited by 3
  32. 2024100% Elimination of Hallucinations on RAGTruth for GPT-4 and GPT-3.5 Turbocited by 2
  33. 2025Do Retrieval-Augmented Language Models Adapt to Varying User Needs?cited by 2
  34. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  35. 2025Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMscited by 2
  36. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generationcited by 2
  37. 2025Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queriescited by 2
  38. 2025CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligencecited by 2

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
  2. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  3. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
  4. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  5. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
  6. 2024Evaluation of Retrieval-Augmented Generation: A Survey
  7. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  8. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions