mlsanityData to 14 Oct 2026

MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation

arXiv 2509.24253 ↗First posted 29 Sep 2025cs.CLcited by 2

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty and ambiguity. We propose MRAG-Suite, a diagnostic evaluation platform integrating diverse multimodal benchmarks (WebQA, Chart-RAG, Visual-RAG, MRAG-Bench). We introduce difficulty-based and ambiguity-aware filtering strategies, alongside MM-RAGChecker, a claim-level diagnostic tool. Our results demonstrate substantial accuracy reductions under difficult and ambiguous queries, highlighting prevalent hallucinations. MM-RAGChecker effectively diagnoses these issues, guiding future improvements in Visual RAG systems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 15 of its 28 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

15 of 28 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation30 May
  2. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents14 Apr
  3. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework20 Feb
  4. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  5. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models10 Oct
  6. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark18 Jul

Other references · 9

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  5. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  6. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  7. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 105
  8. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  9. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluationcited by 9
  2. 2026Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MMEcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
  2. 2026Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generation
  3. 2024Searching for Best Practices in Retrieval-Augmented Generation
  4. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance
  5. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering
  6. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook
  7. 2024RAGViz: Diagnose and Visualize Retrieval-Augmented Generation
  8. 2025A Survey of Multimodal Retrieval-Augmented Generation