# ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks

arXiv: https://arxiv.org/abs/2510.18455 · HTML: https://mlsanity.com/paper/2510.18455/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 24 of 24)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [CRAG -- Comprehensive RAG Benchmark](https://mlsanity.com/paper/2406.04744/) (2024)
- [Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations](https://mlsanity.com/paper/2305.15035/) (2023)
- [OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain](https://mlsanity.com/paper/2412.13018/) (2024)
- [Benchmarking Retrieval-Augmented Generation for Chemistry](https://mlsanity.com/paper/2505.07671/) (2025)
- [LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation](https://mlsanity.com/paper/2502.20640/) (2025)
- [Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models](https://mlsanity.com/paper/2311.08106/) (2023)
- [LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content](https://mlsanity.com/paper/2410.10783/) (2024)
- [A Theory of Dynamic Benchmarks](https://mlsanity.com/paper/2210.03165/) (2022)
- [HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.04800/) (2025)
- [GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?](https://mlsanity.com/paper/2406.05606/) (2024)
- [Evolution of Reddit: From the Front Page of the Internet to a Self-referential Community?](https://mlsanity.com/paper/1402.1386/) (2014)
- [DRAGOn: Designing RAG On Periodically Updated Corpus](https://mlsanity.com/paper/2507.05713/) (2025)

## Abstract (arXiv)

Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics: the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such a benchmark introduces a critical requirement for player-centric authenticity to ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and a dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games to create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Code is avaliable at: https://github.com/hly1998/ChronoPlay.