# Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

arXiv: https://arxiv.org/abs/2510.24870 · HTML: https://mlsanity.com/paper/2510.24870/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 52 of its 89 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 30 of 52)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling](https://mlsanity.com/paper/2501.00574/) (2024)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding](https://mlsanity.com/paper/2411.04952/) (2024)
- [VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos](https://mlsanity.com/paper/2502.01549/) (2025)
- [VideoRAG: Retrieval-Augmented Generation over Video Corpus](https://mlsanity.com/paper/2501.05874/) (2025)
- [FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2311.01477/) (2023)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification](https://mlsanity.com/paper/2406.20079/) (2024)
- [RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering](https://mlsanity.com/paper/2407.13998/) (2024)
- [QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs](https://mlsanity.com/paper/2205.12665/) (2022)
- [VIOLIN: A Large-Scale Dataset for Video-and-Language Inference](https://mlsanity.com/paper/2003.11618/) (2020)
- [VideoXum: Cross-modal Visual and Textural Summarization of Videos](https://mlsanity.com/paper/2303.12060/) (2023)
- [A Closer Look at Claim Decomposition](https://mlsanity.com/paper/2403.11903/) (2024)
- [Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality](https://mlsanity.com/paper/2505.02466/) (2025)
- [Generating Query Focused Summaries from Query-Free Resources](https://mlsanity.com/paper/2012.14774/) (2020)
- [On the Evaluation of Machine-Generated Reports](https://mlsanity.com/paper/2405.00982/) (2024)

## Built on by (most cited first, 1 of 1)

- [Multimodal Fact-Level Attribution for Verifiable Reasoning](https://mlsanity.com/paper/2602.11509/) (2026)

## Abstract (arXiv)

We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a more prevalent source of information online, RAG systems must integrate such media into generation. Yet, existing evaluation methods for RAG are largely text-centric and do not readily transfer to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE and compare it to multimodal variants of three prominent text-centric RAG metrics---ALCE, ARGUE, and RAGAS---finding that MiRAGE outperforms all three on text while being the only one to generalize to multimodal sources. We release open-source implementations and outline evaluation methods for multimodal RAG.