mlsanityData to 7 Oct 2026

UniIR: Training and Benchmarking Universal Multimodal Information Retrievers

arXiv 2311.17136 ↗First posted 28 Nov 2023cs.CV cs.AI cs.CLcited by 165

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image. To approach such different information-seeking demands, we introduce UniIR, a unified instruction-guided multimodal retriever capable of handling eight distinct retrieval tasks across modalities. UniIR, a single retrieval system jointly trained on ten diverse multimodal-IR datasets, interprets user instructions to execute various retrieval tasks, demonstrating robust performance across existing datasets and zero-shot generalization to new tasks. Our experiments highlight that multi-task training and instruction tuning are keys to UniIR's generalization ability. Additionally, we construct the M-BEIR, a multimodal retrieval benchmark with comprehensive results, to standardize the evaluation of universal multimodal information retrieval.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Builds on 53 of its 60 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

53 of 60 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  14. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  16. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  17. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  18. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  19. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  20. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  21. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  22. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 244
  23. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  24. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  25. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  26. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  27. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  28. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 132
  29. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  30. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  31. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  32. 2021WebQA: Multihop and Multimodal QAcited by 109
  33. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  34. 2022Task-aware Retrieval with Instructionscited by 107
  35. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 57
  36. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  37. 2021MURAL: Multimodal, Multitask Retrieval Across Languagescited by 44
  38. 2022Semi-Parametric Neural Image Synthesiscited by 36

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 74; 2026: 61.

165 papers.

Sort:

By area:

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  2. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  3. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  4. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  5. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  6. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  7. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  8. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  9. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  10. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  11. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  12. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmarkcited by 54
38 more
  1. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  2. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  3. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  4. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  5. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  6. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  7. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  8. 2024INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Modelscited by 32
  9. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  10. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  11. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  12. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  13. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Modelscited by 23
  14. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 22
  15. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  16. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  17. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  18. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
  19. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  20. 2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
  21. 2025MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrievalcited by 13
  22. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  23. 2024Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awarenesscited by 12
  24. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  25. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  26. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Modelscited by 12
  27. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  28. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  29. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  30. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  31. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  32. 2024MATE: Meet At The Embedding -- Connecting Images with Long Textscited by 10
  33. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  34. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  35. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  36. 2025ELIP: Enhanced Visual-Language Foundation Models for Image Retrievalcited by 9
  37. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  38. 2024UniRAG: Universal Retrieval Augmentation for Large Vision Language Modelscited by 8

50 most cited of 165 All 165 (JSON)