mlsanityData to 7 Oct 2026

MMDocIR: Benchmarking Multimodal Retrieval for Long Documents

arXiv 2501.08828 ↗First posted 15 Jan 2025cs.IR cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Multimodal document retrieval aims to identify and retrieve various forms of multimodal content, such as figures, tables, charts, and layout information from extensive documents. Despite its increasing popularity, there is a notable lack of a comprehensive and robust benchmark to effectively evaluate the performance of systems in such tasks. To address this gap, this work introduces a new benchmark, named MMDocIR, that encompasses two distinct tasks: page-level and layout-level retrieval. The former evaluates the performance of identifying the most relevant pages within a long document, while the later assesses the ability of detecting specific layouts, providing a more fine-grained measure than whole-page analysis. A layout refers to a variety of elements, including textual paragraphs, equations, figures, tables, or charts. The MMDocIR benchmark comprises a rich dataset featuring 1,685 questions annotated by experts and 173,843 questions with bootstrapped labels, making it a valuable resource in multimodal document retrieval for both training and evaluation. Through rigorous experiments, we demonstrate that (i) visual retrievers significantly outperform their text counterparts, (ii) MMDocIR training set effectively enhances the performance of multimodal document retrieval and (iii) text retrievers leveraging VLM-text significantly outperforms retrievers relying on OCR-text. Our dataset is available at https://mmdocrag.github.io/MMDocIR/.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 38 of its 82 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

38 of 82 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  12. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
26 more
  1. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  2. 2016Image Captioning with Semantic Attentioncited by 260
  3. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  4. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  5. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  6. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  7. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  8. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  9. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 107
  10. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  11. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  12. 2021Document AI: Benchmarks, Models and Applicationscited by 61
  13. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  14. 2017Convolutional Image Captioningcited by 52
  15. 2021Document Collection Visual Question Answeringcited by 45
  16. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  17. 2024DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systemscited by 37
  18. 2024SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluationcited by 23
  19. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 22
  20. 2023From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICLcited by 17
  21. 2024Question-Based Retrieval using Atomic Units for Enterprise RAGcited by 16
  22. 2024Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspectivecited by 12
  23. 2021DocOIE: A Document-level Context-Aware Dataset for OpenIEcited by 10
  24. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  25. 2022Syntactic Multi-view Learning for Open Information Extractioncited by 3
  26. 2014A survey of modern optical character recognition techniquescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 23.

43 papers.

Sort:

By area:

  1. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  2. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
  3. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  4. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 26
  5. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  6. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  7. 2025RAPID: Efficient Retrieval-Augmented Long Text Generation with Writing Planning and Information Discoverycited by 19
  8. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  9. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  10. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  11. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  12. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
31 more
  1. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  2. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  3. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  4. 2026Multi-Vector Index Compression in Any Modalitycited by 8
  5. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  6. 2025CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Qualitycited by 6
  7. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  8. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  9. 2025SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documentscited by 4
  10. 2026Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generationcited by 4
  11. 2026Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testingcited by 3
  12. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Headscited by 3
  13. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 2
  14. 2026MINER: Mining Multimodal Internal Representation for Efficient Retrievalcited by 2
  15. 2026HippoCamp: Benchmarking Contextual Agents on Personal Computerscited by 1
  16. 2026Document-as-Image Representations Fall Short for Scientific Retrievalcited by 1
  17. 2026Very Efficient Listwise Multimodal Reranking for Long Documentscited by 1
  18. 2026From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAGcited by 1
  19. 2025NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuningcited by 0
  20. 2026UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidatescited by 0
  21. 2026Lightweight and Production-Ready PDF Visual Element Parsingcited by 0
  22. 2026Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrievalcited by 0
  23. 2026DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmarkcited by 0
  24. 2026When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrievalcited by 0
  25. 2026PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalizationcited by 0
  26. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  27. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  28. 2026Heterogeneous Element-Aware Cross-Version Differencing of Scientific Documents via Layout-Aware Alignment and Structure-Aware Reasoningcited by 0
  29. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  30. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generationcited by 0
  31. 2026Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework
  2. 2026Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)
  3. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  4. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
  5. 2026MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation
  6. 2026MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval
  7. 2025Docopilot: Improving Multimodal Models for Document-Level Understanding
  8. 2026LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding