mlsanityData to 14 Oct 2026

Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook

arXiv 2503.18016 ↗First posted 23 Mar 2025cs.CVcited by 32

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) has emerged as a pivotal technique in artificial intelligence (AI), particularly in enhancing the capabilities of large language models (LLMs) by enabling access to external, reliable, and up-to-date knowledge sources. In the context of AI-Generated Content (AIGC), RAG has proven invaluable by augmenting model outputs with supplementary, relevant information, thus improving their quality. Recently, the potential of RAG has extended beyond natural language processing, with emerging methods integrating retrieval-augmented strategies into the computer vision (CV) domain. These approaches aim to address the limitations of relying solely on internal model knowledge by incorporating authoritative external knowledge bases, thereby improving both the understanding and generation capabilities of vision models. This survey provides a comprehensive review of the current state of retrieval-augmented techniques in CV, focusing on two main areas: (I) visual understanding and (II) visual generation. In the realm of visual understanding, we systematically review tasks ranging from basic image recognition to complex applications such as medical report generation and multimodal question answering. For visual content generation, we examine the application of RAG in tasks related to image, video, and 3D generation. Furthermore, we explore recent advancements in RAG for embodied AI, with a particular focus on applications in planning, task execution, multimodal perception, interaction, and specialized domains. Given that the integration of retrieval-augmented techniques in CV is still in its early stages, we also highlight the key limitations of current approaches and propose future research directions to drive the development of this promising area.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 111 of its 129 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

111 of 129 references.

Closest ancestors · 27

Same direction, newest first

  1. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries23 Feb
  2. 2025RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning2 Feb
  3. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpus10 Jan
  4. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension20 Nov
  5. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov
  6. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications29 Oct
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  8. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models11 Oct
  9. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models10 Oct
  10. 2024GarmentAligner: Text-to-Garment Generation via Retrieval-augmented Multi-level Corrections22 Aug
  11. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training31 Jul
  12. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning27 Jun
15 more
  1. 2024Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning4 Jun
  2. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning31 May
  3. 2024iRAG: Advancing RAG for Videos with an Incremental Approach18 Apr
  4. 2024Retrieval-Augmented Open-Vocabulary Object Detection8 Apr
  5. 2024Retrieval-Augmented Score Distillation for Text-to-3D Generation5 Feb
  6. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios19 Dec
  7. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension27 Nov
  8. 2023Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation22 Nov
  9. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering29 Sep
  10. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model3 Apr
  11. 2023Retrieval-augmented Image Captioning16 Feb
  12. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledge17 Jan
  13. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec
  14. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct
  15. 2022Retrieval Augmented Classification for Long-Tail Visual Recognition22 Feb

Other references · 84

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
38 more
  1. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  8. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  9. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  10. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  12. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  13. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  14. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  15. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  16. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  17. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  18. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  19. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  20. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  21. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  22. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  23. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  24. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  25. 2021WebQA: Multihop and Multimodal QAcited by 109
  26. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  27. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  28. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 90
  29. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 87
  30. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  31. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  32. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  33. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  34. 2025Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledgecited by 73
  35. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 70
  36. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 60
  37. 2023CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layoutcited by 57
  38. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 14.

32 papers.

Sort:

By area:

  1. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  2. 2025Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?cited by 23
  3. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  4. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 18
  5. 2025Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detectioncited by 15
  6. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 15
  7. 2025Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Methodcited by 13
  8. 2025PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Eracited by 12
  9. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 12
  10. 2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
  11. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 10
  12. 2025WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarioscited by 9
20 more
  1. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  2. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 7
  3. 2025Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Modelcited by 6
  4. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  5. 2026RAID: Retrieval-Augmented Anomaly Detectioncited by 2
  6. 2025Cross-modal RAG: Sub-dimensional Text-to-Image Retrieval-Augmented Generationcited by 1
  7. 2026CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attentioncited by 1
  8. 2025RAG-6DPose: Retrieval-Augmented 6D Pose Estimation via Leveraging CAD as Knowledge Basecited by 0
  9. 2025HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kongcited by 0
  10. 2025Multimedia-Aware Question Answering: A Review of Retrieval and Cross-Modal Reasoning Architecturescited by 0
  11. 2025Seeing the Unseen: Towards Training-Free Inspection for Wind Turbine Blades Using Knowledge-Augmented Vision Language Modelscited by 0
  12. 2025Learning What Helps: Task-Aligned Context Selection for Vision Taskscited by 0
  13. 2026Rationale-Grounded In-Context Learning for Time Series Reasoning with Multimodal Large Language Modelscited by 0
  14. 2026FarmMind: Reasoning-Query-Driven Dynamic Segmentation for Farmland Remote Sensing Imagescited by 0
  15. 2026AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoningcited by 0
  16. 2026Geometric Analysis of Self-Supervised Vision Representations for Semantic Image Retrievalcited by 0
  17. 2026EHR-RAGp: Retrieval-Augmented Prototype-Guided Foundation Model for Electronic Health Recordscited by 0
  18. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  19. 2026ViSAR: Training-Free Adaptive-k Retrieval for Visual Document Question Answeringcited by 0
  20. 2026Visual Search Augmented Chain-of-Thought Reasoning for Attribute Value Extraction from Product Videoscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Searching for Best Practices in Retrieval-Augmented Generation
  2. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  3. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation
  4. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
  5. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelines
  6. 2025Automatic Teaching Platform on Vision Language Retrieval Augmented Generation
  7. 2025CMRAG: Co-modality-based visual document retrieval and question answering
  8. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation