mlsanityData to 14 Oct 2026

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

arXiv 2411.04952 ↗First posted 7 Nov 2024cs.CV cs.AI cs.CLcited by 111

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Document visual question answering (DocVQA) pipelines that answer questions from documents have broad applications. Existing methods focus on handling single-page documents with multi-modal language models (MLMs), or rely on text-based retrieval-augmented generation (RAG) that uses text extraction tools such as optical character recognition (OCR). However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them. We introduce M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions using a multi-modal retriever and an MLM, so that it can efficiently handle single or many documents while preserving visual information. Since previous DocVQA datasets ask questions in the context of a specific document, we also present M3DocVQA, a new benchmark for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages. In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance than many strong baselines, including state-of-the-art performance in MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 56 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

38 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories15 Jun
  2. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov
  3. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
23 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  4. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  6. 2021OCR-free Document Understanding Transformercited by 407
  7. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  9. 2019Scene Text Visual Question Answeringcited by 331
  10. 2024What matters when building vision-language models?cited by 326
  11. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  12. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  13. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  14. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  15. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  16. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  17. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 116
  18. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  19. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 105
  20. 2020Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR)cited by 56
  21. 2023PDFVQA: A New Dataset for Real-World VQA on PDF Documentscited by 28
  22. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  23. 2023Unified Language Representation for Question Answering over Text, Tables, and Imagescited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 37; 2026: 71.

111 papers.

Sort:

By area:

  1. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 72
  2. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  3. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  4. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  5. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  6. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 36
  7. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  8. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  9. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  10. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 28
  11. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 26
  12. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
38 more
  1. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 26
  2. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  3. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  4. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
  5. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  6. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  7. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 16
  8. 2025BookRAG: A Hierarchical Structure-aware Index-based Approach for Retrieval-Augmented Generation on Complex Documentscited by 16
  9. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  10. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  11. 2025MHier-RAG: Multi-Modal RAG for Visual-Rich Document Question-Answering via Hierarchical and Multi-Granularity Reasoningcited by 11
  12. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  13. 2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
  14. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  15. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  16. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Trainingcited by 7
  17. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 7
  18. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  19. 2026DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understandingcited by 6
  20. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  21. 2025URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understandingcited by 5
  22. 2026VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graphcited by 5
  23. 2026AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generationcited by 5
  24. 2026MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learningcited by 5
  25. 2026CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligencecited by 5
  26. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  27. 2026R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generationcited by 4
  28. 2026MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translationcited by 4
  29. 2026Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQAcited by 4
  30. 2026PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMscited by 4
  31. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  32. 2025SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understandingcited by 3
  33. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 3
  34. 2025CogDoc: Towards Unified thinking in Documentscited by 3
  35. 2026Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchainscited by 3
  36. 2026MoDora: Tree-Based Semi-Structured Document Analysis Systemcited by 3
  37. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  38. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 3

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation
  2. 2025Enhancing Document VQA Models via Retrieval-Augmented Generation
  3. 2026Multimodal Graph RAG for Long-range Visually Rich Document Understanding
  4. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
  5. 2025PDF Retrieval Augmented Question Answering
  6. 2025Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation
  7. 2026MultiDocFusion: Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documents
  8. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents