mlsanityData to 14 Oct 2026

Adapting MLLMs for Nuanced Video Retrieval

arXiv 2512.13511 ↗First posted 15 Dec 2025cs.CV cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Our objective is to build an embedding model that captures the nuanced relationship between a search query and candidate videos. We cover three aspects of nuanced retrieval: (i) temporal, (ii) negation, and (iii) multimodal. For temporal nuance, we consider chiral actions that need distinguishing between temporally opposite actions like "opening a door" vs. "closing a door". For negation, we consider queries with negators such as "not", "none" that allow a user to specify what they do not want. For multimodal nuance, we consider the task of composed retrieval where the query comprises a video along with a text edit instruction. The goal is to develop a unified embedding model that handles such nuances effectively. To that end, we repurpose a Multimodal Large Language Model (MLLM) trained to generate text into an embedding model. We fine-tune it with a contrastive loss on text alone with carefully sampled hard negatives that instill the desired nuances in the learned embedding space. Despite the text-only training, our method achieves state of the art performance on all benchmarks for nuanced video retrieval. We also analyze how this improvement is achieved, and show that text-only training reduces the modality gap between text and video embeddings leading to better organization of the embedding space.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 62 of its 80 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

62 of 80 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
38 more
  1. 2017Localizing Moments in Video with Natural Languagecited by 663
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  4. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  5. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  7. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  8. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  9. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  10. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  11. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  12. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  13. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  14. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  15. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  16. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  17. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  18. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  19. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
  20. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  21. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  22. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  23. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  24. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  25. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  26. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  27. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  28. 2025Vision-Language Models Do Not Understand Negationcited by 72
  29. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  30. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  31. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  32. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  33. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  34. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 42
  35. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 41
  36. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 30
  37. 2019Action Modifiers: Learning from Adverbs in Instructional Videoscited by 27
  38. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27

50 most cited of 62 All 62 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VeRVE: Versatile Retrieval for Videos via Unified Embeddings
  2. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
  3. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
  4. 2025RzenEmbed: Towards Comprehensive Multimodal Retrieval
  5. 2024NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval
  6. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrieval
  7. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction
  8. 2026MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?