mlsanityData to 7 Oct 2026

Evaluation of Retrieval-Augmented Generation: A Survey

arXiv 2405.07437 ↗First posted 13 May 2024cs.CL cs.AIcited by 146

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 56 of its 74 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

56 of 74 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models17 Apr
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey29 Feb
  3. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation19 Feb
  4. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  5. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  6. 2024The Power of Noise: Redefining Retrieval for RAG Systems26 Jan
  7. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System11 Jan
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  9. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  10. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep
  11. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation28 Sep

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2017Billion-scale similarity search with GPUscited by 2,477
33 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  8. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  9. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  12. 2022RealTime QA: What's the Answer Right Now?cited by 239
  13. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  14. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  15. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  16. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  17. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  18. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  19. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
  20. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  21. 2018EventKG: A Multilingual Event-Centric Temporal Knowledge Graphcited by 43
  22. 2023Merging Generated and Retrieved Knowledge for Open-Domain QAcited by 36
  23. 2023Learning to Reason and Memorize with Self-Notescited by 30
  24. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28
  25. 2023UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generationcited by 22
  26. 2024Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agentscited by 22
  27. 2024Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmarkcited by 22
  28. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  29. 2021Understanding Jargon: Combining Extraction and Generation for Definition Modelingcited by 15
  30. 2023Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboardcited by 15
  31. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  32. 2022Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysiscited by 10
  33. 2023On the Opportunities of Green Computing: A Surveycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 77; 2026: 40.

146 papers.

Sort:

By area:

  1. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  2. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  3. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  4. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 113
  5. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  6. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 51
  7. 2025In-depth Analysis of Graph-based RAG in a Unified Frameworkcited by 45
  8. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  9. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  10. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  11. 2025KET-RAG: A Cost-Efficient Multi-Granular Indexing Framework for Graph-RAGcited by 29
  12. 2025ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generationcited by 27
38 more
  1. 2025Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generationcited by 23
  2. 2024Adaptive Retrieval-Augmented Generation for Conversational Systemscited by 22
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  4. 2025DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answeringcited by 20
  5. 2024Multi-Head RAG: Solving Multi-Aspect Problems with LLMscited by 18
  6. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Surveycited by 18
  7. 2024Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalabilitycited by 15
  8. 2025CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answeringcited by 15
  9. 2025Who Stole Your Data? A Method for Detecting Unauthorized RAG Theftcited by 15
  10. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  11. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  12. 2025Credible Plan-Driven RAG Method for Multi-Hop Question Answeringcited by 12
  13. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  14. 2025Beyond the Crowd: LLM-Augmented Community Notes for Governing Health Misinformationcited by 10
  15. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 9
  16. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  17. 2025CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMscited by 8
  18. 2025Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factualitycited by 8
  19. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspectivecited by 7
  20. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systemscited by 7
  21. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  22. 2024A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validationcited by 6
  23. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  24. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  25. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  26. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generationcited by 5
  27. 2024GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answeringcited by 5
  28. 2024Enterprise Benchmarks for Large Language Model Evaluationcited by 5
  29. 2025Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysiscited by 5
  30. 2025Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agentscited by 5
  31. 2025DRAGOn: Designing RAG On Periodically Updated Corpuscited by 5
  32. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  33. 2026CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoningcited by 5
  34. 2024HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settingscited by 4
  35. 2024Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needscited by 4
  36. 2024Unanswerability Evaluation for Retrieval Augmented Generationcited by 4
  37. 2025ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systemscited by 4
  38. 2025Enhancing Domain-Specific Retrieval-Augmented Generation: Synthetic Data Generation and Evaluation using Reasoning Modelscited by 4

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  2. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  3. 2024Retrieval-Augmented Generation for Natural Language Processing: A Survey
  4. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  5. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  6. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generation
  7. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  8. 2025Dynamic and Parametric Retrieval-Augmented Generation