mlsanityData to 7 Oct 2026

Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey

arXiv 2601.03262 ↗First posted 16 Dec 2025cs.IR cs.CLcited by 2

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Visually rich documents (VRDs) challenge retrieval-augmented generation (RAG) with layout-dependent semantics, brittle OCR, and evidence spread across complex figures and structured tables. This survey examines how Multimodal Large Language Models (MLLMs) are being used to make VRD retrieval practical for RAG. We organize the literature into three roles: Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers. We compare these roles along retrieval granularity, information fidelity, latency and index size, and compatibility with reranking and grounding. We also outline key trade-offs and offer some practical guidance on when to favor each role. Finally, we identify promising directions for future research, including adaptive retrieval units, model size reduction, and the development of evaluation methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 93 of its 125 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

93 of 125 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval2 Nov
  2. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG4 Oct
  3. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding10 Aug
  4. 2025MHier-RAG: Multi-Modal RAG for Visual-Rich Document Question-Answering via Hierarchical and Multi-Granularity Reasoning1 Aug
  5. 2025SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding9 Jun
  6. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents14 Apr
  7. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook23 Mar
  8. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts24 Feb
  9. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework20 Feb
  10. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos3 Feb
  11. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpus10 Jan
  12. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance7 Jan
4 more
  1. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension20 Nov
  2. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications29 Oct
  3. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  4. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
38 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  13. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  14. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  15. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  16. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  17. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  18. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  19. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  20. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  21. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  22. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  23. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  24. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  25. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  26. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  27. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  28. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  29. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  30. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  31. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  32. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  33. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  34. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  35. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  36. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  37. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  38. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  2. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
  2. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelines
  3. 2025A Multi-Granularity Retrieval Framework for Visually-Rich Documents
  4. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  5. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
  6. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
  7. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
  8. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generation