# Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering

arXiv: https://arxiv.org/abs/2505.16470 · HTML: https://mlsanity.com/paper/2505.16470/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 30 of 38)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder](https://mlsanity.com/paper/2205.12035/) (2022)
- [ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation](https://mlsanity.com/paper/2407.06135/) (2024)
- [M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding](https://mlsanity.com/paper/2411.04952/) (2024)
- [Unifying Multimodal Retrieval via Document Screenshot Embedding](https://mlsanity.com/paper/2406.11251/) (2024)
- [Hierarchical multimodal transformers for Multi-Page DocVQA](https://mlsanity.com/paper/2212.05935/) (2022)
- [Towards Complex Document Understanding By Discrete Reasoning](https://mlsanity.com/paper/2207.11871/) (2022)
- [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://mlsanity.com/paper/2401.10208/) (2024)
- [Learning Customized Visual Models with Retrieval-Augmented Knowledge](https://mlsanity.com/paper/2301.07094/) (2023)

## Built on by (most cited first, 30 of 36)

- [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://mlsanity.com/paper/2505.18761/) (2025)
- [DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding](https://mlsanity.com/paper/2511.11552/) (2025)
- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [DocDancer: Towards Agentic Document-Grounded Information Seeking](https://mlsanity.com/paper/2601.05163/) (2026)
- [Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation](https://mlsanity.com/paper/2508.15709/) (2025)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory](https://mlsanity.com/paper/2605.15710/) (2026)
- [Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing](https://mlsanity.com/paper/2601.04554/) (2026)
- [Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation](https://mlsanity.com/paper/2604.10741/) (2026)
- [Multimodal Fact-Level Attribution for Verifiable Reasoning](https://mlsanity.com/paper/2602.11509/) (2026)
- [MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation](https://mlsanity.com/paper/2606.26458/) (2026)
- [GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents](https://mlsanity.com/paper/2607.11192/) (2026)
- [Benchmarking Deflection and Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2604.12033/) (2026)
- [Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge](https://mlsanity.com/paper/2604.22939/) (2026)
- [Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2606.26079/) (2026)
- [Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence](https://mlsanity.com/paper/2609.32041/) (2026)
- [NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2510.18940/) (2025)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)
- [Lightweight and Production-Ready PDF Visual Element Parsing](https://mlsanity.com/paper/2604.23276/) (2026)
- [DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark](https://mlsanity.com/paper/2605.30027/) (2026)
- [When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval](https://mlsanity.com/paper/2606.01304/) (2026)
- [Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing](https://mlsanity.com/paper/2606.01393/) (2026)
- [MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios](https://mlsanity.com/paper/2606.14747/) (2026)
- [MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering](https://mlsanity.com/paper/2607.01420/) (2026)
- [CMDR: Contextual Multimodal Document Retrieval](https://mlsanity.com/paper/2607.05927/) (2026)
- [Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction](https://mlsanity.com/paper/2608.08459/) (2026)
- [HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation](https://mlsanity.com/paper/2608.14032/) (2026)
- [What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering](https://mlsanity.com/paper/2608.14841/) (2026)
- [Question-Guided Evidence Acquisition for Multimodal Visual Question Answering](https://mlsanity.com/paper/2608.19739/) (2026)
- [DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?](https://mlsanity.com/paper/2608.26757/) (2026)

## Abstract (arXiv)

Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents (text, images, tables) and performing cross-modal reasoning. Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches, frequently missing critical visual information. The field also lacks robust benchmarks for assessing multimodal evidence selection and integration. We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains. Our framework introduces innovative metrics for evaluating multimodal quote selection and enables answers that interleave text with relevant visual elements. Through large-scale experiments with 60 VLM/LLM models and 14 retrieval systems, we identify persistent challenges in multimodal evidence retrieval, selection, and integration.Key findings reveal advanced proprietary LVMs show superior performance than open-sourced alternatives. Also, they show moderate advantages using multimodal inputs over text-only inputs, while open-source alternatives show significant performance degradation. Notably, fine-tuned LLMs achieve substantial improvements when using detailed image descriptions. MMDocRAG establishes a rigorous testing ground and provides actionable insights for developing more robust multimodal DocVQA systems. Our benchmark and code are available at https://mmdocrag.github.io/MMDocRAG/.