mlsanityData to 7 Oct 2026

LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning

arXiv 2503.04812 ↗First posted 4 Mar 2025cs.CV cs.AI cs.CLcited by 54

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Universal multimodal embedding models play a critical role in tasks such as interleaved image-text retrieval, multimodal RAG, and multimodal clustering. However, our empirical results indicate that existing LMM-based embedding models trained with the standard InfoNCE loss exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. To deal with this issue, we propose a simple yet effective framework that dynamically improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. Within this framework, we train a series of models, named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks and 36 datasets. Experimental results show that LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance while demonstrating strong scalability and efficiency. Specifically, LLaVE-2B surpasses the previous SOTA 7B models, while LLaVE-7B achieves a further performance improvement of 6.2 points. Although LLaVE is trained on image-text data, it can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance, demonstrating its remarkable potential for transfer to other embedding tasks.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

27 of 47 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2020Contrastive Learning with Hard Negative Samplescited by 619
15 more
  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  2. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  3. 2020Self-supervised Graph Learning for Recommendationcited by 305
  4. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  5. 2019Contrastive Learning of Structured World Modelscited by 206
  6. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  7. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  8. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  9. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  10. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  11. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  12. 2020Are all negatives created equal in contrastive instance discrimination?cited by 50
  13. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  14. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularitycited by 9
  15. 2023Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCEcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 25.

54 papers.

Sort:

By area:

  1. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  2. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  3. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  4. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  5. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  6. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  7. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  8. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  9. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  10. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  11. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 21
  12. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
38 more
  1. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  2. 2025Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrievalcited by 12
  3. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  4. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  5. 2025Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculumcited by 9
  6. 2025Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrievalcited by 9
  7. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  8. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  9. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  10. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 6
  11. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  12. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  13. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  14. 2025LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Modelscited by 5
  15. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  16. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  17. 2026Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddingscited by 5
  18. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  19. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrievalcited by 4
  20. 2026CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehensioncited by 4
  21. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  22. 2025Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimizationcited by 3
  23. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  24. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  25. 2026Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecturecited by 3
  26. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  27. 2026V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrievalcited by 2
  28. 2026Pailitao-VL: Unified Embedding and Reranker for Real-Time Multi-Modal Industrial Searchcited by 2
  29. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 2
  30. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrievalcited by 2
  31. 2026Efficient and High-Fidelity Omni Modality Retrievalcited by 2
  32. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?cited by 1
  33. 2026mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrievalcited by 1
  34. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoningcited by 1
  35. 2026m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encodercited by 1
  36. 2026Illuminating Visual Identity in Universal Multimodal Embeddingscited by 1
  37. 2025GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learningcited by 0
  38. 2025FANoise: Singular Value-Adaptive Noise Modulation for Robust Multimodal Representation Learningcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
  2. 2025Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding
  3. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
  4. 2026Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
  5. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  6. 2026Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
  7. 2024E5-V: Universal Embeddings with Multimodal Large Language Models
  8. 2025xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation