mlsanityData to 14 Oct 2026

Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering

arXiv 2505.16470 ↗First posted 22 May 2025cs.IR cs.CL cs.CVcited by 36

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents (text, images, tables) and performing cross-modal reasoning. Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches, frequently missing critical visual information. The field also lacks robust benchmarks for assessing multimodal evidence selection and integration. We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains. Our framework introduces innovative metrics for evaluating multimodal quote selection and enables answers that interleave text with relevant visual elements. Through large-scale experiments with 60 VLM/LLM models and 14 retrieval systems, we identify persistent challenges in multimodal evidence retrieval, selection, and integration.Key findings reveal advanced proprietary LVMs show superior performance than open-sourced alternatives. Also, they show moderate advantages using multimodal inputs over text-only inputs, while open-source alternatives show significant performance degradation. Notably, fine-tuned LLMs achieve substantial improvements when using detailed image descriptions. MMDocRAG establishes a rigorous testing ground and provides actionable insights for developing more robust multimodal DocVQA systems. Our benchmark and code are available at https://mmdocrag.github.io/MMDocRAG/.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 70 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

38 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov
  2. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications29 Oct
  3. 2024MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering16 Aug
  4. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledge17 Jan
  5. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
21 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  5. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  6. 2021InfographicVQAcited by 437
  7. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  8. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  9. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  10. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  11. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  12. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 110
  13. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 105
  14. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  15. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  16. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  17. 2024DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systemscited by 37
  18. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  19. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  20. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 31
  21. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 31.

36 papers.

Sort:

By area:

  1. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 28
  2. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  3. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  4. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 12
  5. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
  6. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  7. 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memorycited by 6
  8. 2026Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testingcited by 4
  9. 2026Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generationcited by 4
  10. 2026Multimodal Fact-Level Attribution for Verifiable Reasoningcited by 2
  11. 2026MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generationcited by 2
  12. 2026GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documentscited by 2
24 more
  1. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Modelscited by 1
  2. 2026Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledgecited by 1
  3. 2026Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Modelscited by 1
  4. 2026Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligencecited by 1
  5. 2025NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuningcited by 0
  6. 2026Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysiscited by 0
  7. 2026Lightweight and Production-Ready PDF Visual Element Parsingcited by 0
  8. 2026DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmarkcited by 0
  9. 2026When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrievalcited by 0
  10. 2026Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsingcited by 0
  11. 2026MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarioscited by 0
  12. 2026MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answeringcited by 0
  13. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  14. 2026Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Constructioncited by 0
  15. 2026HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generationcited by 0
  16. 2026What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answeringcited by 0
  17. 2026Question-Guided Evidence Acquisition for Multimodal Visual Question Answeringcited by 0
  18. 2026DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?cited by 0
  19. 2026Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agentscited by 0
  20. 2026Collaborative Memory for Multi-Agent VLM Systemscited by 0
  21. 2026Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QAcited by 0
  22. 2026TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoningcited by 0
  23. 2026Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QAcited by 0
  24. 2026MMKG-R1: Constructing Multimodal Knowledge Graphs via Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation
  2. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
  3. 2026MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
  4. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
  5. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  6. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
  7. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
  8. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding