mlsanityData to 7 Oct 2026

VIRTUE: Visual-Interactive Text-Image Universal Embedder

arXiv 2510.00523 ↗First posted 1 Oct 2025cs.AI cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multimodal representation learning models have demonstrated successful operation across complex tasks, and the integration of vision-language models (VLMs) has further enabled embedding models with instruction-following capabilities. However, existing embedding models lack visual-interactive capabilities to specify regions of interest from users (e.g., point, bounding box, mask), which have been explored in generative models to broaden their human-interactive applicability. Equipping embedding models with visual interactions not only would unlock new applications with localized grounding of user intent, which remains unexplored, but also enable the models to learn entity-level information within images to complement their global representations for conventional embedding tasks. In this paper, we propose a novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends the capabilities of the segmentation model and the vision-language model to the realm of representation learning. In VIRTUE, the segmentation model can process visual prompts that pinpoint specific regions within an image, thereby enabling the embedder to handle complex and ambiguous scenarios more precisely. To evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples that aims to retrieve the text caption by jointly considering the entity with a specific object and image scene. VIRTUE consistently achieves a state-of-the-art performance with significant improvements across 36 universal MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 59 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

45 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs24 Apr
  2. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant2 Dec
  3. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct
  4. 2024E5-V: Universal Embeddings with Multimodal Large Language Models17 Jul

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Learning without Forgettingcited by 2,465
29 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  5. 2021Grounded Language-Image Pre-trainingcited by 1,147
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  13. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  14. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  15. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  16. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  17. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  18. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  19. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  20. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  21. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  22. 2025VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thoughtcited by 47
  23. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  24. 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
  25. 2021You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Predictioncited by 23
  26. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  27. 2024Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrievalcited by 11
  28. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMscited by 7
  29. 2023ShuttleSHAP: A Turn-Based Feature Attribution Approach for Analyzing Forecasting Models in Badmintoncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  2. 2026Omni-Interactive Universal Embeddercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  2. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
  3. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
  4. 2024OLIVE: Object Level In-Context Visual Embeddings
  5. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
  6. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
  7. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Model
  8. 2026UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editing