Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
Not on the map. Why?
Abstract (arXiv)
With the rapid proliferation of multimodal information, Visual Document Retrieval (VDR) has emerged as a critical frontier in bridging the gap between unstructured visually rich data and precise information acquisition. Unlike traditional natural image retrieval, visual documents exhibit unique characteristics defined by dense textual content, intricate layouts, and fine-grained semantic dependencies. This paper surveys the VDR landscape as a retrieval problem in its own right, rather than as the front end of a generation pipeline, and does so specifically through the lens of the Multimodal Large Language Model (MLLM) era. We begin by examining the benchmark landscape, including the recent turn toward reasoning-intensive evaluation, and then dive into the methodological evolution along two orthogonal axes: what a retriever is, spanning multimodal embedding models and reranker models, and how it is deployed, from single-stage retrieval through Retrieval-Augmented Generation (RAG) to Agentic systems. Cutting across both, we analyse the representation--efficiency trade-off that late interaction imposes. We ground these categories in leaderboard evidence on where the empirical frontier actually lies, and close by identifying persistent challenges and outlining promising future directions for multimodal document intelligence.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 180 of its 230 references that are in the map. First posted on arXiv 23 Feb 2026.
Builds on
180 of 230 references.
References · 180
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
- 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
- 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
- 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
- 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
- 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
- 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
- 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
- 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
- 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
38 more
- 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
- 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
- 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
- 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
- 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
- 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 120
- 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
- 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
- 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 107
- 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
- 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
- 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
- 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
- 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
- 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
- 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
- 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
- 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
- 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 69
- 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
- 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
- 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
- 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
- 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
- 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 66
- 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 64
- 2025Deep Research: A Survey of Autonomous Research Agentscited by 63
- 2021Document AI: Benchmarks, Models and Applicationscited by 61
- 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
- 2025ViDoRe Benchmark V2: Raising the Bar for Visual Retrievalcited by 58
- 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
- 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
- 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
- 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
- 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
- 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
- 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
- 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
50 most cited of 180 All 180 (JSON)
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
- 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
- 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Frameworkcited by 2
- 2026PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generationcited by 2
- 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrievalcited by 1
- 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrievalcited by 0
- 2026KoViDoRe: Korean Visual Document Retrievalcited by 0
- 2026Spatial Matryoshka Training for Multi-Granularity Visual Document Retrievalcited by 0
- 2026Spatial Matryoshka Training for Multi-Granularity Visual Document Retrieval30 Aug 2026
- 2026KoViDoRe: Korean Visual Document Retrieval21 Aug 2026
- 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval18 Aug 2026
- 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrieval9 Aug 2026
- 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework3 Jun 2026
- 2026PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation1 Jun 2026
- 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval11 Apr 2026
- 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations2 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2025A Multi-Granularity Retrieval Framework for Visually-Rich Documents
- 2026MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval
- 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval
- 2025Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction
- 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
- 2026HIVE: Query, Hypothesize, Verify An LLM Framework for Multimodal Reasoning-Intensive Retrieval
- 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
- 2026DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding