mlsanityData to 14 Oct 2026

Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

arXiv 2411.19710 ↗First posted 29 Nov 2024cs.IR cs.LGcited by 8

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval Augmented Generation (RAG) systems are a widespread application of Large Language Models (LLMs) in the industry. While many tools exist empowering developers to build their own systems, measuring their performance locally, with datasets reflective of the system's use cases, is a technological challenge. Solutions to this problem range from non-specific and cheap (most public datasets) to specific and costly (generating data from local documents). In this paper, we show that using public question and answer (Q&A) datasets to assess retrieval performance can lead to non-optimal systems design, and that common tools for RAG dataset generation can lead to unbalanced data. We propose solutions to these issues based on the characterization of RAG datasets through labels and through label-targeted data generation. Finally, we show that fine-tuned small LLMs can efficiently generate Q&A datasets. We believe that these observations are invaluable to the know-your-data step of RAG systems development.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 65 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

28 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers22 Mar
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
13 more
  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  5. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  7. 2020How Context Affects Language Models' Factual Predictionscited by 181
  8. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  9. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 113
  10. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  11. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  12. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  13. 2024FaaF: Facts as a Function for the evaluation of generated textcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluationcited by 8
  2. 2025A Survey on Retrieval And Structuring Augmented Generation with Large Language Modelscited by 8
  3. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluationcited by 3
  4. 2026Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluationcited by 2
  5. 2025Talking to Data: Designing Smart Assistants for Humanities Databasescited by 0
  6. 2026The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG Systemcited by 0
  7. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systemscited by 0
  8. 2026Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  2. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  4. 2025Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana
  5. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  6. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation System
  7. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  8. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective