mlsanityData to 7 Oct 2026

RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

arXiv 2408.01262 ↗First posted 2 Aug 2024cs.CL cs.IRcited by 39

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) is a powerful approach that enables large language models (LLMs) to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance to evaluate LLM generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 60 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

25 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RAG Does Not Work for Enterprises31 May
  2. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  3. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  5. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 20

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  10. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
8 more
  1. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  3. 2024Large Language Models: A Surveycited by 506
  4. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  5. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  6. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  7. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  8. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 23; 2026: 10.

39 papers.

Sort:

By area:

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  3. 2025Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaborationcited by 19
  4. 2025FinSage: A Multi-aspect RAG System for Financial Filings Question Answeringcited by 18
  5. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  6. 2024Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responsescited by 15
  7. 2025MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generationcited by 15
  8. 2024Do We Need Domain-Specific Embedding Models? An Empirical Investigationcited by 12
  9. 2025DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidencecited by 10
  10. 2025XRAG: Cross-lingual Retrieval-Augmented Generationcited by 9
  11. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 9
  12. 2024Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Studycited by 8
27 more
  1. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  2. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  3. 2025UltraRAG: A Modular and Automated Toolkit for Adaptive Retrieval-Augmented Generationcited by 6
  4. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generationcited by 5
  5. 2024RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systemscited by 4
  6. 2025Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generationscited by 4
  7. 2025ALoFTRAG: Automatic Local Fine Tuning for Retrieval Augmented Generationcited by 3
  8. 2025A System for Comprehensive Assessment of RAG Frameworkscited by 2
  9. 2025Benchmarking Vector, Graph and Hybrid Retrieval Augmented Generation (RAG) Pipelines for Open Radio Access Networks (ORAN)cited by 2
  10. 2025CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generationcited by 2
  11. 2025Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queriescited by 2
  12. 2025An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generationcited by 1
  13. 2025RAST: A Retrieval Augmented Spatio-Temporal Framework for Traffic Predictioncited by 1
  14. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAGcited by 1
  15. 2026Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Modelscited by 1
  16. 2026Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiabilitycited by 1
  17. 2025AI-Driven Climate Policy Scenario Generation for Sub-Saharan Africacited by 0
  18. 2025Unlocking the Potential of Large Language Models in the Nuclear Industry with Synthetic Datacited by 0
  19. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generationcited by 0
  20. 2026Injecting Knowledge from Social Science Journals to Improve Indonesian Cultural Understanding by LLMscited by 0
  21. 2026Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AIcited by 0
  22. 2026RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corporacited by 0
  23. 2026Privacy Without Losing Place: A Paradigm for Private Retrieval in Spatial RAGscited by 0
  24. 2026MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrievalcited by 0
  25. 2026Quantifying Prior Dominance in RAG Systemscited by 0
  26. 2026RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Modelscited by 0
  27. 2026EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  2. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  3. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
  4. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  5. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  6. 2024InspectorRAGet: An Introspection Platform for RAG Evaluation
  7. 2026LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
  8. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework