mlsanityData to 7 Oct 2026

VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

arXiv 2504.09795 ↗First posted 14 Apr 2025cs.CL cs.AI cs.CVcited by 57

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually-rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 58 of its 70 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

58 of 70 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024VISA: Retrieval Augmented Generation with Visual Source Attribution19 Dec
  2. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov
  3. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  4. 2024Retrieval-Augmented Score Distillation for Text-to-3D Generation5 Feb
  5. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model3 Apr
  6. 2023Retrieval-augmented Image Captioning16 Feb
  7. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov
  8. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generator29 Sep

Other references · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  13. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  14. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  15. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  16. 2019Scene Text Visual Question Answeringcited by 327
  17. 2023Improving Text Embeddings with Large Language Modelscited by 295
  18. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  19. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  20. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  21. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  22. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  23. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  24. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  25. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  26. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  27. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  28. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 107
  29. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  30. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  31. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  32. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  33. 2023Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Tablecited by 40
  34. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  35. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  36. 2024Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Modelscited by 19
  37. 2023Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboardcited by 15
  38. 2023MPMQA: Multimodal Question Answering on Product Manualscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 41.

57 papers.

Sort:

By area:

  1. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 27
  2. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 21
  3. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  4. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  5. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  6. 2025RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understandingcited by 8
  7. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  8. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generationcited by 5
  9. 2025HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documentscited by 5
  10. 2026AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generationcited by 5
  11. 2026CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligencecited by 5
  12. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
38 more
  1. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  2. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4
  3. 2026MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learningcited by 4
  4. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  5. 2025SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understandingcited by 3
  6. 2025TextlessRAG: End-to-End Visual Document RAG by Speech Without Textcited by 2
  7. 2025MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generationcited by 2
  8. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understandingcited by 2
  9. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  10. 2026RAID: Retrieval-Augmented Anomaly Detectioncited by 2
  11. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  12. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  13. 2026PSG-Nav: Probabilistic Scene Graph Navigation via Multiverse Decision Makingcited by 2
  14. 2026MAGE-RAG: Multigranular Adaptive Graph Evidence for Agentic Multimodal RAG in Long-Document QAcited by 2
  15. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 2
  16. 2026EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrievalcited by 2
  17. 2025CReSt: A Comprehensive Benchmark for Retrieval-Augmented Generation with Complex Reasoning over Structured Documentscited by 1
  18. 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrievalcited by 1
  19. 2026Exploring Information Seeking Agent Consolidationcited by 0
  20. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAGcited by 0
  21. 2026LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrievalcited by 0
  22. 2026Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Modelcited by 0
  23. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  24. 2026Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answeringcited by 0
  25. 2026LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understandingcited by 0
  26. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  27. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  28. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  29. 2026VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documentscited by 0
  30. 2026XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understandingcited by 0
  31. 2026KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrievalcited by 0
  32. 2026DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understandingcited by 0
  33. 2026Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attributioncited by 0
  34. 2026InSight-doc: Agentic Visual Perception for Long-Document Understandingcited by 0
  35. 2026D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understandingcited by 0
  36. 2026Question-Guided Evidence Acquisition for Multimodal Visual Question Answeringcited by 0
  37. 2026Case2Flow: Bridging Patient Cases and Guideline Flowcharts through Multimodal Retrievalcited by 0
  38. 2026UniGeo: A Multi-modal Large Language Model for Text-Guided Cross-View Geo-Localizationcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Document VQA Models via Retrieval-Augmented Generation
  2. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
  3. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
  4. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents
  5. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
  6. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
  7. 2025CMRAG: Co-modality-based visual document retrieval and question answering
  8. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering