mlsanityData to 14 Oct 2026

Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

arXiv 2404.15406 ↗First posted 23 Apr 2024cs.CV cs.AI cs.CLcited by 80

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 47 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

43 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
30 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2017Billion-scale similarity search with GPUscited by 2,497
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  13. 2023Active Retrieval Augmented Generationcited by 516
  14. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  15. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  16. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  17. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  18. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  19. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  20. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  21. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  22. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  23. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  24. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  25. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  26. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  27. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  28. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  29. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  30. 2024Grounding Language Models for Visual Entity Recognitioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 31; 2026: 27.

80 papers.

Sort:

By area:

  1. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  2. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  3. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  4. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 43
  5. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
  6. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  8. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  9. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Modelscited by 23
  10. 2024GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-trainingcited by 23
  11. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 20
  12. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
38 more
  1. 2025CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAGcited by 18
  2. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  3. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 16
  4. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  5. 2025Towards General Continuous Memory for Vision-Language Modelscited by 15
  6. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 15
  7. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  8. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  9. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  10. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  11. 2024Retrieval Augmented Diffusion Model for Structure-informed Antibody Design and Optimizationcited by 9
  12. 2024Towards Retrieval-Augmented Architectures for Image Captioningcited by 7
  13. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  14. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMscited by 6
  15. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answeringcited by 6
  16. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  17. 2024Multimodal Music Generation with Explicit Bridges and Retrieval Augmentationcited by 6
  18. 2025SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Modelcited by 6
  19. 2024Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimizationcited by 5
  20. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  21. 2025ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrievalcited by 5
  22. 2026ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answeringcited by 5
  23. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottleneckscited by 4
  24. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  25. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4
  26. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 4
  27. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  28. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  29. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  30. 2026CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answeringcited by 2
  31. 2026Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoningcited by 2
  32. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  33. 2024Composing Open-domain Vision with RAG for Ocean Monitoring and Conservationcited by 1
  34. 2025Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Modelscited by 1
  35. 2025Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environmentcited by 1
  36. 2025MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recyclingcited by 1
  37. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  38. 2025Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solvingcited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
  2. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  3. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
  4. 2025Evaluating Multimodal Large Language Models on Educational Textbook Question Answering
  5. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
  6. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
  7. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  8. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering