mlsanityData to 7 Oct 2026

Evaluating Retrieval Quality in Retrieval-Augmented Generation

arXiv 2404.13781 ↗First posted 21 Apr 2024cs.CL cs.IRcited by 90

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's τ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 36 references that are in the map. First posted on arXiv 21 Apr 2024.

Builds on

21 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation9 Apr
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  10. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  11. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  12. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
7 more
  1. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  2. 2023Large language models can accurately predict searcher preferencescited by 129
  3. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  4. 2022Retrieval-Enhanced Machine Learningcited by 46
  5. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  6. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10
  7. 2023An Offline Metric for the Debiasedness of Click Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 36; 2026: 31.

90 papers.

Sort:

By area:

  1. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
  2. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  3. 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 35
  4. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  5. 2024Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximizationcited by 29
  6. 2025Traceback of Poisoning Attacks to Retrieval-Augmented Generationcited by 28
  7. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  8. 2024Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Modelscited by 23
  9. 2024Adaptive Retrieval-Augmented Generation for Conversational Systemscited by 22
  10. 2024A Survey of Conversational Searchcited by 22
  11. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  12. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
38 more
  1. 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
  2. 2025Rethinking Data Protection in the (Generative) Artificial Intelligence Eracited by 17
  3. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
  4. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generationcited by 15
  5. 2025LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Sciencecited by 15
  6. 2024Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximizationcited by 14
  7. 2024Accelerating Retrieval-Augmented Generationcited by 14
  8. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systemscited by 11
  9. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  10. 2025Is Relevance Propagated from Retriever to Generator in RAG?cited by 11
  11. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  12. 2024Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversationscited by 9
  13. 2025MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answeringcited by 9
  14. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  15. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspectivecited by 7
  16. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  17. 2025RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practicecited by 6
  18. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  19. 2024Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysiscited by 5
  20. 2025Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysiscited by 5
  21. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 5
  22. 2026PruneRAG: Confidence-Guided Query Decomposition Trees for Efficient Retrieval-Augmented Generationcited by 5
  23. 2024Toward Optimal Search and Retrieval for RAGcited by 4
  24. 2025ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systemscited by 4
  25. 2025Enhancing Domain-Specific Retrieval-Augmented Generation: Synthetic Data Generation and Evaluation using Reasoning Modelscited by 4
  26. 2025LeRAAT: LLM-Enabled Real-Time Aviation Advisory Toolcited by 4
  27. 2025LTRR: Learning To Rank Retrievers for LLMscited by 4
  28. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  29. 2024Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scorescited by 3
  30. 2024Retrieval-Augmented Generation for Mobile Edge Computing via Large Language Modelcited by 3
  31. 2026Diagnosing and Repairing Citation Failures in Generative Engine Optimizationcited by 3
  32. 2026Learning to Retrieve from Agent Trajectoriescited by 3
  33. 2024Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Taskscited by 2
  34. 2025How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAGcited by 2
  35. 2025TransLaw: A Large-Scale Dataset and Multi-Agent Benchmark Simulating Professional Translation of Hong Kong Case Lawcited by 2
  36. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2
  37. 2026Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generationcited by 2
  38. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systemscited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluation of Retrieval-Augmented Generation: A Survey
  2. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  3. 2024ARAGOG: Advanced RAG Output Grading
  4. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
  5. 2024Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data
  6. 2025Redefining Retrieval Evaluation in the Era of LLMs
  7. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  8. 2025Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control