mlsanityData to 7 Oct 2026

RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

arXiv 2408.08067 ↗First posted 15 Aug 2024cs.CL cs.AIcited by 59

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems. This work has been open sourced at https://github.com/amazon-science/RAGChecker.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 57 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

34 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  2. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models17 Apr
  3. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  4. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  6. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2024Mixtral of Expertscited by 1,785
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  6. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  7. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  8. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  9. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  10. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  11. 2023Improving Text Embeddings with Large Language Modelscited by 295
  12. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
16 more
  1. 2021Internet-Augmented Dialogue Generationcited by 217
  2. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 217
  3. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  4. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  5. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  6. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  7. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  8. 2023Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Modelscited by 67
  9. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  10. 2023Retrieval-Augmented Code Generation for Universal Information Extractioncited by 40
  11. 2023Evaluating Open-QA Evaluationcited by 35
  12. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  13. 2021Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialogcited by 33
  14. 2024RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Modelscited by 24
  15. 2024Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmarkcited by 22
  16. 2024Prompt-based Code Completion via Multi-Retrieval Augmented Generationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 37.

59 papers.

Sort:

By area:

  1. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  2. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  3. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  4. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
  5. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  6. 2025End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoningcited by 10
  7. 2024LOGO -- Long cOntext aliGnment via efficient preference Optimizationcited by 9
  8. 2025Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factualitycited by 8
  9. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systemscited by 7
  10. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  11. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  12. 2026When Is Enough Not Enough? Illusory Completion in Search Agentscited by 5
38 more
  1. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 5
  2. 2025Bridging Industrial Expertise and XR with LLM-Powered Conversational Agentscited by 4
  3. 2025Metadata-Driven Retrieval-Augmented Generation for Financial Question Answeringcited by 4
  4. 2026Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorialcited by 4
  5. 2026Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agentscited by 4
  6. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
  7. 2025Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generationcited by 3
  8. 2025MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluationcited by 3
  9. 2025OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviewscited by 3
  10. 2024Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generationcited by 2
  11. 2025Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generationcited by 2
  12. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  13. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2
  14. 2026RAID: Retrieval-Augmented Anomaly Detectioncited by 2
  15. 2026Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understandingcited by 2
  16. 2026CUE-R: Beyond the Final Answer in Retrieval-Augmented Generationcited by 2
  17. 2026Fine-grained Claim-level RAG Benchmark for Lawcited by 2
  18. 2026Evaluating Chunking Strategies For Retrieval-Augmented Generation in Oil and Gas Enterprise Documentscited by 1
  19. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  20. 2026Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?cited by 1
  21. 2026MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&Acited by 1
  22. 2026A Failure-Mode Benchmark for Polymorphic Sybil Poisoning in RAGcited by 1
  23. 2026Bridging the Question-Answer Gap in Retrieval-Augmented Generation: Hypothetical Prompt Embeddingscited by 1
  24. 2026Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiabilitycited by 1
  25. 2025CRAB: A Benchmark for Evaluating Curation of Retrieval-Augmented LLMs in Biomedicinecited by 0
  26. 2025Maximally-Informative Retrieval for State Space Model Generationcited by 0
  27. 2025SPAR: Session-based Pipeline for Adaptive Retrieval on Legacy File Systemscited by 0
  28. 2026BLUEmed: Retrieval-Augmented Multi-Agent Debate for Clinical Error Detectioncited by 0
  29. 2026ShadowMerge: A Novel Poisoning Attack on Graph-Based Agent Memory via Relation-Channel Conflictscited by 0
  30. 2026Persistent and Conversational Multi-Method Explainability for Trustworthy Financial AIcited by 0
  31. 2026Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)cited by 0
  32. 2026Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Workcited by 0
  33. 2026BioDivergence: A Benchmark and Evaluation Framework for Hidden Contextual Contradictions in Biomedical Abstractscited by 0
  34. 2026X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generationcited by 0
  35. 2026Retrieval-Warmed Energy-Based Reasoning: A Five-Arm Ablation Methodology for Diffusion-as-Inference on Structured Reasoning Taskscited by 0
  36. 2026Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitationscited by 0
  37. 2026How Much Do Legal RAG Systems Still Hallucinate?cited by 0
  38. 2026Evidence Blindness in Direct Corpus Interaction: Persistent Navigation with AtlasNavcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation
  2. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
  3. 2025OmniBench-RAG: A Multi-Domain Evaluation Platform for Retrieval-Augmented Generation Tools
  4. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  5. 2024InspectorRAGet: An Introspection Platform for RAG Evaluation
  6. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  7. 2025Engineering the RAG Stack: A Comprehensive Review of the Architecture and Trust Frameworks for Retrieval-Augmented Generation Systems
  8. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning