mlsanityData to 7 Oct 2026

E5-V: Universal Embeddings with Multimodal Large Language Models

arXiv 2407.12580 ↗First posted 17 Jul 2024cs.CL cs.CV cs.IRcited by 123

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings. Our findings highlight the significant potential of MLLMs in representing multimodal inputs compared to previous approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning. We propose a single modality training approach for E5-V, where the model is trained exclusively on text pairs. This method demonstrates significant improvements over traditional multimodal training on image-text pairs, while reducing training costs by approximately 95%. Additionally, this approach eliminates the need for costly multimodal training data collection. Extensive experiments across four types of tasks demonstrate the effectiveness of E5-V. As a universal multimodal model, E5-V not only achieves but often surpasses state-of-the-art performance in each task, despite being trained on a single modality.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 30 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

27 of 30 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
15 more
  1. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  2. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  3. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  4. 2023Improving Text Embeddings with Large Language Modelscited by 295
  5. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 244
  6. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  7. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  8. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  9. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  10. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  11. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  12. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  13. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 57
  14. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  15. 2024iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrievalcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 52; 2026: 60.

123 papers.

Sort:

By area:

  1. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  2. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  3. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  4. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  5. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  6. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  7. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  8. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  9. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  10. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  11. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  12. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
38 more
  1. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  2. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  3. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  4. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  5. 2025CoLLM: A Large Language Model for Composed Image Retrievalcited by 21
  6. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 21
  7. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  8. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  9. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  10. 2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 18
  11. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  12. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  13. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  14. 2025MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrievalcited by 13
  15. 2025Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrievalcited by 12
  16. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  17. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  18. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  19. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  20. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  21. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  22. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  23. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  24. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  25. 2026TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrievalcited by 7
  26. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  27. 2025WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMcited by 6
  28. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  29. 2024A Survey of Multimodal Composite Editing and Retrievalcited by 5
  30. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  31. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  32. 2026Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddingscited by 5
  33. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  34. 2025FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representationscited by 4
  35. 2025From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videoscited by 4
  36. 2025SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documentscited by 4
  37. 2026VeRVE: Versatile Retrieval for Videos via Unified Embeddingscited by 4
  38. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrievalcited by 4

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
  2. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  6. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  8. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques