mlsanityData to 14 Oct 2026

RzenEmbed: Towards Comprehensive Multimodal Retrieval

arXiv 2510.27350 ↗First posted 31 Oct 2025cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

The rapid advancement of Multimodal Large Language Models (MLLMs) has extended CLIP-based frameworks to produce powerful, universal embeddings for retrieval tasks. However, existing methods primarily focus on natural images, offering limited support for other crucial visual modalities such as videos and visual documents. To bridge this gap, we introduce RzenEmbed, a unified framework to learn embeddings across a diverse set of modalities, including text, images, videos, and visual documents. We employ a novel two-stage training strategy to learn discriminative representations. The first stage focuses on foundational text and multimodal retrieval. In the second stage, we introduce an improved InfoNCE loss, incorporating two key enhancements. Firstly, a hardness-weighted mechanism guides the model to prioritize challenging samples by assigning them higher weights within each batch. Secondly, we implement an approach to mitigate the impact of false negatives and alleviate data noise. This strategy not only enhances the model's discriminative power but also improves its instruction-following capabilities. We further boost performance with learnable temperature parameter and model souping. RzenEmbed sets a new state-of-the-art on the MMEB benchmark. It not only achieves the best overall score but also outperforms all prior work on the challenging video and visual document retrieval tasks. Our models are available in https://huggingface.co/qihoo360/RzenEmbed.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 51 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

47 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model1 Aug
  2. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying28 May
  3. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs24 Apr
  4. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs4 Nov
  5. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct
  6. 2024ColPali: Efficient Document Retrieval with Vision Language Models27 Jun

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
29 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  10. 2020Contrastive Learning with Hard Negative Samplescited by 628
  11. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  12. 2023Improving Text Embeddings with Large Language Modelscited by 303
  13. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  14. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  15. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  16. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  17. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  18. 2022Curriculum Temperature for Knowledge Distillationcited by 112
  19. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  20. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  21. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  22. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
  23. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  24. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  25. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  26. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  27. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 28
  28. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  29. 2023Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualizationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 29.

29 papers.

Sort:

By area:

  1. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  2. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 10
  3. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  4. 2026WeMM-Embedding: WeChat Multi-Modal Embedding Technical Reportcited by 8
  5. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 7
  6. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  7. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  8. 2026MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Controlcited by 6
  9. 2026LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneckcited by 5
  10. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  11. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
  12. 2026V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrievalcited by 2
17 more
  1. 2026Bottleneck Tokens for Unified Multimodal Retrievalcited by 2
  2. 2026ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuningcited by 2
  3. 2026Douyin Multimodal Embedding Model Technical Reportcited by 2
  4. 2026UEmbed: Unified Sparse and Dense Multimodal Embeddingscited by 2
  5. 2026Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editingcited by 1
  6. 2026OSCAR: Optimization-Steered Agentic Planning for Composed Image Retrievalcited by 1
  7. 2026Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matchingcited by 1
  8. 2026FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrievalcited by 0
  9. 2026Do Composed Image Retrieval Benchmarks Require Multimodal Composition?cited by 0
  10. 2026MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarioscited by 0
  11. 2026Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrievalcited by 0
  12. 2026DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Groundingcited by 0
  13. 2026RegRet: Enhancing Region-Level Retrieval in Large Multimodal Modelscited by 0
  14. 2026VaME: Exploring Variational Latent Reasoning for Multimodal Embeddingscited by 0
  15. 2026CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understandingcited by 0
  16. 2026Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillationcited by 0
  17. 2026A Simulation-Grounded Agentic VLM Framework for Wildfire Monitoring and Reportingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction
  2. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  3. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
  4. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?
  5. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval
  6. 2025Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video
  7. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
  8. 2026ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression