# DRAGOn: Designing RAG On Periodically Updated Corpus

arXiv: https://arxiv.org/abs/2507.05713 · HTML: https://mlsanity.com/paper/2507.05713/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 12 of 12)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)
- [MERA: A Comprehensive LLM Evaluation in Russian](https://mlsanity.com/paper/2401.04531/) (2024)
- [The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design](https://mlsanity.com/paper/2408.12503/) (2024)
- [RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues](https://mlsanity.com/paper/2409.12558/) (2024)
- [Facilitating large language model Russian adaptation with Learned Embedding Propagation](https://mlsanity.com/paper/2412.21140/) (2024)

## Built on by (most cited first, 5 of 5)

- [Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries](https://mlsanity.com/paper/2510.11956/) (2025)
- [ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks](https://mlsanity.com/paper/2510.18455/) (2025)
- [Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels](https://mlsanity.com/paper/2512.07474/) (2025)
- [OCC-RAG: Optimal Cognitive Core for Faithful Question Answering](https://mlsanity.com/paper/2606.00683/) (2026)
- [Little Brains, Big Feats: Exploring Compact Language Models](https://mlsanity.com/paper/2606.30062/) (2026)

## Abstract (arXiv)

This paper introduces DRAGOn, method to design a RAG benchmark on a regularly updated corpus. It features recent reference datasets, a question generation framework, an automatic evaluation pipeline, and a public leaderboard. Specified reference datasets allow for uniform comparison of RAG systems, while newly generated dataset versions mitigate data leakage and ensure that all models are evaluated on unseen, comparable data. The pipeline for automatic question generation extracts the Knowledge Graph from the text corpus and produces multiple question-answer pairs utilizing modern LLM capabilities. A set of diverse LLM-as-Judge metrics is provided for a comprehensive model evaluation. We used Russian news outlets to form the datasets and demonstrate our methodology. We launch a public leaderboard to track the development of RAG systems and encourage community participation.