# Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana

arXiv: https://arxiv.org/abs/2501.12789 · HTML: https://mlsanity.com/paper/2501.12789/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- First posted on arXiv 22 Jan 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Built on by (most cited first, 13 of 13)

- [LTRR: Learning To Rank Retrievers for LLMs](https://mlsanity.com/paper/2506.13743/) (2025)
- [RAGentA: Multi-Agent Retrieval-Augmented Generation for Attributed Question Answering](https://mlsanity.com/paper/2506.16988/) (2025)
- [Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.21384/) (2025)
- [Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge](https://mlsanity.com/paper/2506.22644/) (2025)
- [CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training](https://mlsanity.com/paper/2506.10844/) (2025)
- [Knowledge-Aware Diverse Reranking for Cross-Source Question Answering](https://mlsanity.com/paper/2506.20476/) (2025)
- [CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks](https://mlsanity.com/paper/2606.03650/) (2026)
- [DoTA-RAG: Dynamic of Thought Aggregation RAG](https://mlsanity.com/paper/2506.12571/) (2025)
- [RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition](https://mlsanity.com/paper/2506.14412/) (2025)
- [UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses](https://mlsanity.com/paper/2506.22210/) (2025)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)
- [How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation](https://mlsanity.com/paper/2606.12789/) (2026)
- [SIGIR 2025 -- LiveRAG Challenge Report](https://mlsanity.com/paper/2507.04942/) (2025)

## Abstract (arXiv)

Evaluating Retrieval-Augmented Generation (RAG) systems, especially in domain-specific contexts, requires benchmarks that address the distinctive requirements of the applicative scenario. Since real data can be hard to obtain, a common strategy is to use LLM-based methods to generate synthetic data. Existing solutions are general purpose: given a document, they generate a question to build a Q&A pair. However, although the generated questions can be individually good, they are typically not diverse enough to reasonably cover the different ways real end-users can interact with the RAG system. We introduce here DataMorgana, a tool for generating highly customizable and diverse synthetic Q&A benchmarks tailored to RAG applications. DataMorgana enables detailed configurations of user and question categories and provides control over their distribution within the benchmark. It uses a lightweight two-stage process, ensuring efficiency and fast iterations, while generating benchmarks that reflect the expected traffic. We conduct a thorough line of experiments, showing quantitatively and qualitatively that DataMorgana surpasses existing tools and approaches in producing lexically, syntactically, and semantically diverse question sets across domain-specific and general-knowledge corpora. DataMorgana will be made available to selected teams in the research community, as first beta testers, in the context of the upcoming SIGIR'2025 LiveRAG challenge to be announced in early February 2025.