mlsanityData to 7 Oct 2026

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

arXiv 2411.02571 ↗First posted 4 Nov 2024cs.CL cs.AI cs.CVcited by 159

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

State-of-the-art retrieval models typically address a straightforward search scenario, in which retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but it underperforms compared to a smaller CLIP retriever in cross-modal retrieval tasks due to the modality bias exhibited by MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose continuously fine-tuning the universal multimodal retriever to enhance its text retrieval capability while preserving multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on the MTEB retrieval benchmark. We also explore prompting the off-the-shelf MLLMs as zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that, through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way for advancing universal multimodal retrieval in the future.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 65 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

52 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024E5-V: Universal Embeddings with Multimodal Large Language Models17 Jul
  2. 2024ColPali: Efficient Document Retrieval with Vision Language Models27 Jun

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
38 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  6. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  7. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  9. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  10. 2023Improving Text Embeddings with Large Language Modelscited by 295
  11. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  12. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  13. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 244
  14. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  15. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  16. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  17. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  18. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 132
  19. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  20. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  21. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  22. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  23. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  24. 2021WebQA: Multihop and Multimodal QAcited by 109
  25. 2022Task-aware Retrieval with Instructionscited by 107
  26. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 107
  27. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
  28. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  29. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  30. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 57
  31. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 53
  32. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  33. 2019Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedbackcited by 42
  34. 2024PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrievalcited by 30
  35. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  36. 2023EDIS: Entity-Driven Image Search over Multimodal Web Contentcited by 27
  37. 2023Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answeringcited by 18
  38. 2024MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipelinecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 66; 2026: 90.

159 papers.

Sort:

By area:

  1. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  2. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  3. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  4. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  5. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  6. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  7. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  8. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  9. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  10. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  11. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 36
  12. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
38 more
  1. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 31
  2. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  3. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  4. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  5. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  6. 2025Continual Multimodal Contrastive Learningcited by 19
  7. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  8. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  9. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  10. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  11. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  12. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  13. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  14. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  15. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  16. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  17. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  18. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  19. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  20. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  21. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  22. 2025MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understandingcited by 7
  23. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  24. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  25. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  26. 2026TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrievalcited by 7
  27. 2025Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Upcited by 6
  28. 2025MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularizationcited by 6
  29. 2025WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMcited by 6
  30. 2025Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Rerankingcited by 6
  31. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  32. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  33. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  34. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  35. 2025M3Retrieve: Benchmarking Multimodal Retrieval for Medicinecited by 5
  36. 2025MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understandingcited by 5
  37. 2026Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddingscited by 5
  38. 2026Gemini Embedding 2: A Native Multimodal Embedding Model from Geminicited by 5

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
  2. 2026Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval
  3. 2025Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval
  4. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning
  5. 2025Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation
  6. 2025DocMMIR: A Framework for Document Multi-modal Information Retrieval
  7. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?
  8. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search