mlsanityData to 14 Oct 2026

VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents

arXiv 2410.10594 ↗First posted 14 Oct 2024cs.IR cs.AI cs.CLcited by 245

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) is an effective technique that enables large language models (LLMs) to utilize external knowledge sources for generation. However, current RAG systems are solely based on text, rendering it impossible to utilize vision information like layout and images that play crucial roles in real-world multi-modality documents. In this paper, we introduce VisRAG, which tackles this issue by establishing a vision-language model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the document to obtain text, the document is directly embedded using a VLM as an image and then retrieved to enhance the generation of a VLM. Compared to traditional text-based RAG, VisRAG maximizes the retention and utilization of the data information in the original documents, eliminating the information loss introduced during the parsing process. We collect both open-source and synthetic data to train the retriever in VisRAG and explore a variety of generation methods. Experiments demonstrate that VisRAG outperforms traditional RAG in both the retrieval and generation stages, achieving a 20--40% end-to-end performance gain over traditional text-based RAG pipeline. Further analysis reveals that VisRAG is efficient in utilizing training data and demonstrates strong generalization capability, positioning it as a promising solution for RAG on multi-modality documents. Our code and data are available at https://github.com/openbmb/visrag.

Key facts. Cited by 245 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 67 of its 78 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

67 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning31 May
  2. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  13. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  14. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  16. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  17. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  18. 2023Active Retrieval Augmented Generationcited by 516
  19. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  20. 2021InfographicVQAcited by 437
  21. 2021OCR-free Document Understanding Transformercited by 407
  22. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  23. 2024What matters when building vision-language models?cited by 326
  24. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  25. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  26. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  27. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  28. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  29. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  30. 2024Corrective Retrieval Augmented Generationcited by 197
  31. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  32. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  33. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  34. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  35. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  36. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 151
  37. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  38. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 134

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 94; 2026: 144.

245 papers.

Sort:

By area:

  1. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  2. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  3. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  4. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 77
  5. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 72
  6. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  7. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  8. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  9. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  10. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  11. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 42
  12. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 38
38 more
  1. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  2. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  3. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 35
  4. 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Modelscited by 34
  5. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  6. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  7. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  8. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  9. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 26
  10. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  11. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 26
  12. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  13. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 23
  14. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 23
  15. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 21
  16. 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 21
  17. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  18. 2025RAG-Anything: All-in-One RAG Frameworkcited by 20
  19. 2025Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challengescited by 19
  20. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  21. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 18
  22. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  23. 2025Towards General Continuous Memory for Vision-Language Modelscited by 15
  24. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  25. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 14
  26. 2026Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memorycited by 14
  27. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  28. 2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contextscited by 12
  29. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  30. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 12
  31. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 12
  32. 2025Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Searchcited by 11
  33. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 11
  34. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 10
  35. 2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
  36. 2025Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?cited by 10
  37. 2026Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrievalcited by 10
  38. 2025SuperRAG: Beyond RAG with Layout-Aware Graph Modelingcited by 9

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelines
  2. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey
  3. 2024Searching for Best Practices in Retrieval-Augmented Generation
  4. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  5. 2025A Multi-Granularity Retrieval Framework for Visually-Rich Documents
  6. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  7. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework
  8. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities