mlsanityData to 7 Oct 2026

UniRAG: Universal Retrieval Augmentation for Large Vision Language Models

arXiv 2405.10311 ↗First posted 16 May 2024cs.IRcited by 8

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Recently, Large Vision Language Models (LVLMs) have unlocked many complex use cases that require Multi-Modal (MM) understanding (e.g., image captioning or visual question answering) and MM generation (e.g., text-guided image generation or editing) capabilities. To further improve the output fidelityof LVLMs we introduce UniRAG, a plug-and-play technique that adds relevant retrieved information to prompts as few-shot examples during inference. Unlike the common belief that Retrieval Augmentation (RA) mainly improves generation or understanding of uncommon entities, our evaluation results on the MSCOCO dataset with common entities show that both proprietary models like GPT-4o and Gemini-Pro and smaller open-source models like LLaVA, LaVIT, and Emu2 significantly enhance their generation quality when their input prompts are augmented with relevant information retrieved by Vision-Language (VL) retrievers like UniIR models. All the necessary code to reproduce our results is available at https://github.com/castorini/UniRAG

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 47 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

29 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
16 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  5. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  6. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  8. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  11. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  12. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  13. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answeringcited by 97
  14. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  15. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  16. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 4.

8 papers.

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  2. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  3. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  4. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  5. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  6. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  7. 2024RaSeRec: Retrieval-Augmented Sequential Recommendationcited by 3
  8. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge
  2. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
  3. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learning
  4. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
  5. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models
  6. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning