mlsanityData to 14 Oct 2026

Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

arXiv 2510.24870 ↗First posted 28 Oct 2025cs.CL cs.CV cs.IRcited by 1

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a more prevalent source of information online, RAG systems must integrate such media into generation. Yet, existing evaluation methods for RAG are largely text-centric and do not readily transfer to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE and compare it to multimodal variants of three prominent text-centric RAG metrics---ALCE, ARGUE, and RAGAS---finding that MiRAGE outperforms all three on text while being the only one to generalize to multimodal sources. We release open-source implementations and outline evaluation methods for multimodal RAG.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 52 of its 89 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

52 of 89 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025WikiVideo: Article Generation from Multiple Videos1 Apr
  2. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos3 Feb
  3. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpus10 Jan
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov

Other references · 48

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  5. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2017Dense-Captioning Events in Videoscited by 793
  8. 2020Contrastive Learning with Hard Negative Samplescited by 628
  9. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  10. 2019ELI5: Long Form Question Answeringcited by 575
  11. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  12. 2019Clotho: An Audio Captioning Datasetcited by 358
36 more
  1. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  3. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
  4. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  5. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Modelscited by 46
  6. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 46
  7. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  8. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  9. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  10. 2020VIOLIN: A Large-Scale Dataset for Video-and-Language Inferencecited by 41
  11. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  12. 2024A Closer Look at Claim Decompositioncited by 41
  13. 2025Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modalitycited by 30
  14. 2020Generating Query Focused Summaries from Query-Free Resourcescited by 27
  15. 2024On the Evaluation of Machine-Generated Reportscited by 23
  16. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 23
  17. 2024Audio Entailment: Assessing Deductive Reasoning for Audio Understandingcited by 22
  18. 2024DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generationcited by 22
  19. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  20. 2025Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrievalcited by 20
  21. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
  22. 2024Overview of the TREC 2023 NeuCLIR Trackcited by 17
  23. 2024MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrievalcited by 17
  24. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answeringcited by 17
  25. 2024Core: Robust Factual Precision with Informative Sub-Claim Identificationcited by 16
  26. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognitioncited by 14
  27. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  28. 2025Auto-ARGUE: LLM-Based Report Generation Evaluationcited by 14
  29. 2023A Unified View of Evaluation Metrics for Structured Predictioncited by 11
  30. 2024Grounding Partially-Defined Events in Multimodal Datacited by 10
  31. 2023MegaWika: Millions of reports and their sources across 50 diverse languagescited by 9
  32. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  33. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  34. 2025MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusioncited by 8
  35. 2024Overview of TREC 2024 Biomedical Generative Retrieval (BioGen) Trackcited by 7
  36. 2024Every Answer Matters: Evaluating Commonsense with Probabilistic Measurescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Multimodal Fact-Level Attribution for Verifiable Reasoningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
  2. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelines
  3. 2024Searching for Best Practices in Retrieval-Augmented Generation
  4. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage
  5. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  6. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  7. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation
  8. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG