mlsanityData to 7 Oct 2026

Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup

arXiv 2403.00820 ↗First posted 26 Feb 2024cs.IR cs.CLcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

also in 1 more
Abstract (arXiv)

Retrieval Augmented Generation (RAG) systems have seen huge popularity in augmenting Large-Language Model (LLM) outputs with domain specific and time sensitive data. Very recently a shift is happening from simple RAG setups that query a vector database for additional information with every user input to more sophisticated forms of RAG. However, different concrete approaches compete on mostly anecdotal evidence at the moment. In this paper we present a rigorous dataset creation and evaluation workflow to quantitatively compare different RAG strategies. We use a dataset created this way for the development and evaluation of a boolean agent RAG setup: A system in which a LLM can decide whether to query a vector database or not, thus saving tokens on questions that can be answered with internal knowledge. We publish our code and generated dataset online.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 11 of its 14 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

11 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generator15 Jan
  2. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  6. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  7. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  8. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  9. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 54

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agentcited by 46
  2. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  2. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  4. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  5. 2024Semantic Tokens in Retrieval Augmented Generation
  6. 2026Is Agentic RAG worth it? An experimental comparison of RAG approaches
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey
  8. 2024Deploying Large Language Models With Retrieval Augmented Generation