mlsanityData to 7 Oct 2026

VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents

arXiv 2507.04590 ↗First posted 7 Jul 2025cs.CV cs.CLcited by 110

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

45 of 63 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2017Dense-Captioning Events in Videoscited by 789
33 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  3. 2017Localizing Moments in Video with Natural Languagecited by 660
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  5. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  7. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  8. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  9. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  10. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  11. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  12. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  13. 2023Improving Text Embeddings with Large Language Modelscited by 295
  14. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  15. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  16. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  17. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  18. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  19. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  20. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  21. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  22. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  23. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  24. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  25. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  26. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  27. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  28. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 64
  29. 2025ViDoRe Benchmark V2: Raising the Bar for Visual Retrievalcited by 58
  30. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  31. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  32. 2025Uni-Retrieval: A Multi-Style Retrieval Framework for STEM's Educationcited by 13
  33. 2024UniRAG: Universal Retrieval Augmentation for Large Vision Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 91.

110 papers.

Sort:

By area:

  1. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  2. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  3. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  4. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  5. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  6. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  7. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  8. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  9. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  10. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddingscited by 12
  11. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  12. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
38 more
  1. 2025Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculumcited by 9
  2. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  3. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  4. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  5. 2025WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMcited by 6
  6. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 6
  7. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  8. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  9. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 6
  10. 2026MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Controlcited by 6
  11. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
  12. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  13. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  14. 2026VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graphcited by 5
  15. 2026Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddingscited by 5
  16. 2026LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneckcited by 5
  17. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  18. 2025Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embeddingcited by 4
  19. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrievalcited by 4
  20. 2026CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehensioncited by 4
  21. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 4
  22. 2026WeMM-Embedding: WeChat Multi-Modal Embedding Technical Reportcited by 4
  23. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  24. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  25. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  26. 2026Reasoning-Augmented Representations for Multimodal Retrievalcited by 3
  27. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
  28. 2026End-to-End Semantic ID Generation for Generative Advertisement Recommendationcited by 3
  29. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  30. 2026Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecturecited by 3
  31. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 3
  32. 2026TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokenscited by 3
  33. 2026Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audiocited by 3
  34. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  35. 2026PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrievalcited by 2
  36. 2026Efficient and High-Fidelity Omni Modality Retrievalcited by 2
  37. 2026Bottleneck Tokens for Unified Multimodal Retrievalcited by 2
  38. 2026Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrievalcited by 2

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  2. 2026VeRVE: Versatile Retrieval for Videos via Unified Embeddings
  3. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
  4. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
  5. 2026Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
  6. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
  7. 2025Enhancing Subsequent Video Retrieval via Vision-Language Models (VLMs)
  8. 2026VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models