mlsanityData to 14 Oct 2026

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

arXiv 2410.05160 ↗First posted 7 Oct 2024cs.CV cs.AI cs.CLcited by 228

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite its importance and practicality. In this work, we aim to explore the potential for building universal embeddings capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training and 16 evaluation datasets covering both in-distribution and out-of-distribution tasks, and (2) VLM2Vec (Vision-Language Model -> Vector), a contrastive training framework that converts any state-of-the-art vision-language model into an embedding model via training on MMEB. Unlike previous models such as CLIP and BLIP, which encodes text or images independently without any task instruction, VLM2Vec can process any combination of images and text to generate a fixed-dimensional vector based on task instructions. We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split. Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in MMEB. We show that VLMs are secretly strong embedding models.

Key facts. Cited by 228 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 81 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

67 of 81 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024E5-V: Universal Embeddings with Multimodal Large Language Models17 Jul

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  13. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  14. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  15. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  16. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  17. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  18. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  21. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  22. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  23. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  24. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  25. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  26. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  27. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  28. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  29. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  30. 2021InfographicVQAcited by 437
  31. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  32. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  33. 2016Visual Dialogcited by 377
  34. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  35. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  36. 2015Skip-Thought Vectorscited by 322
  37. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  38. 2023Improving Text Embeddings with Large Language Modelscited by 303

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 79; 2026: 142.

228 papers.

Sort:

By area:

  1. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  2. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  3. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  4. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  5. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  6. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  7. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  8. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  9. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  10. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  11. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 41
  12. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 39
38 more
  1. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 38
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  4. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 29
  5. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 28
  6. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  7. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  8. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 23
  9. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 22
  10. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 21
  11. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 21
  12. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  13. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  14. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
  15. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  16. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  17. 2025MIEB: Massive Image Embedding Benchmarkcited by 15
  18. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 14
  19. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  20. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddingscited by 13
  21. 2025Closing the Modality Gap for Mixed Modality Searchcited by 12
  22. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  23. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  24. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 11
  25. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 11
  26. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  27. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  28. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 10
  29. 2026Multi-Vector Index Compression in Any Modalitycited by 10
  30. 2025CLaMR: Contextualized Late-Interaction for Multimodal Content Retrievalcited by 9
  31. 2025Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystemscited by 9
  32. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  33. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  34. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  35. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 8
  36. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  37. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  38. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 8

50 most cited of 228 All 228 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
  2. 2026LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  4. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  5. 2025Towards General Continuous Memory for Vision-Language Models
  6. 2022A Survey of Vision-Language Pre-Trained Models
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  8. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval