# MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation

arXiv: https://arxiv.org/abs/2601.15487 · HTML: https://mlsanity.com/paper/2601.15487/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 18 of its 78 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)

## Builds on (most cited first, 18 of 18)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [MoreHopQA: More Than Multi-hop Reasoning](https://mlsanity.com/paper/2406.13397/) (2024)
- [Nomic Embed Vision: Expanding the Latent Space](https://mlsanity.com/paper/2406.18587/) (2024)
- [MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation](https://mlsanity.com/paper/2504.17137/) (2025)
- [WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts](https://mlsanity.com/paper/2506.15594/) (2025)
- [Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluation](https://mlsanity.com/paper/2409.04475/) (2024)
- [WeQA: A Benchmark for Retrieval Augmented Generation in Wind Energy Domain](https://mlsanity.com/paper/2408.11800/) (2024)
- [How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models](https://mlsanity.com/paper/2508.16757/) (2025)

## Built on by (most cited first, 1 of 1)

- [CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets](https://mlsanity.com/paper/2610.15409/) (2026)

## Abstract (arXiv)

The rapid evolution of Retrieval-Augmented Generation (RAG) toward multimodal, high-stakes enterprise applications has outpaced the development of domain specific evaluation benchmarks. Existing datasets often rely on general-domain corpora or purely textual retrieval, failing to capture the complexity of specialized technical documents where information is inextricably multimodal and reasoning requires synthesizing disjoint evidence. We address this gap by introducing MiRAGE, a Multiagent framework for RAG systems Evaluation, that leverages a collaborative swarm of specialized agents to generate verified, domain-specific, multimodal, and multi-hop Question-Answer datasets. MiRAGE orchestrates a swarm of specialized agents: a recursive context optimization loop to aggregate scattered evidence, an adversarial verifier agent to guarantee factual grounding, and an agent to recognize the expert persona and the relevant domain to mimic expert cognitive workflows. Extensive empirical evaluation across four distinct domains (regulations, finance, quantitative biology, and journalism) demonstrates that MiRAGE generates datasets with significantly higher reasoning complexity (>2.3 average hops) and factual faithfulness. Our ablation studies point that MiRAGE can be powered by LLMs if textual descriptions of the images are available. Visual grounding still remains a frontier. By automating the creation of gold standard evaluation datasets that reflect the latent thematic structure of proprietary corpora, MiRAGE provides the necessary infrastructure to rigorously benchmark the next generation information retrieval systems.