mlsanityData to 14 Oct 2026

CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity

arXiv 2410.12248 ↗First posted 16 Oct 2024cs.CLcited by 11

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) aims to enhance large language models (LLMs) to generate more accurate and reliable answers with the help of the retrieved context from external knowledge sources, thereby reducing the incidence of hallucinations. Despite the advancements, evaluating these systems remains a crucial research area due to the following issues: (1) Limited data diversity: The insufficient diversity of knowledge sources and query types constrains the applicability of RAG systems; (2) Obscure problems location: Existing evaluation methods have difficulty in locating the stage of the RAG pipeline where problems occur; (3) Unstable retrieval evaluation: These methods often fail to effectively assess retrieval performance, particularly when the chunking strategy changes. To tackle these challenges, we propose a Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline, including chunking, retrieval, reranking, and generation. To effectively evaluate the first three phases, we introduce multi-granularity keywords, including coarse-grained and fine-grained keywords, to assess the retrieved context instead of relying on the annotation of golden chunks. Moreover, we release a holistic benchmark dataset tailored for diverse data scenarios covering a wide range of document formats and query types. We demonstrate the utility of the CoFE-RAG framework by conducting experiments to evaluate each stage of RAG systems. Our evaluation method provides unique insights into the effectiveness of RAG systems in handling diverse data scenarios, offering a more nuanced understanding of their capabilities and limitations.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 13 of its 22 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

13 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 11

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  3. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  5. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  6. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  7. 2022Large Language Models with Controllable Working Memorycited by 169
  8. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  9. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  10. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  11. 2024Piccolo2: General Text Embedding with Multi-task Hybrid Loss Trainingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  2. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 10
  3. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  4. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  5. 2025RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelinescited by 4
  6. 2025Multi-Hop Question Generation via Dual-Perspective Keyword Guidancecited by 3
  7. 2025LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraintscited by 3
  8. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAGcited by 1
  9. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategiescited by 1
  10. 2026Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)cited by 0
  11. 2026Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  2. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  3. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  4. 2024Evaluation of Retrieval-Augmented Generation: A Survey
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  6. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
  7. 2025RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation
  8. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning