mlsanityData to 14 Oct 2026

Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval

arXiv 2604.25273 ↗First posted 28 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Despite significant progress in Unified Multimodal Retrieval (UMR) powered by Large Multimodal Models (LMMs), existing embedding methods primarily focus on sample-level objectives via contrastive learning while overlooking the crucial subject-level semantics. This limitation hinders the model's ability to group semantically coherent subjects in complex multimodal queries, manifesting as semantic alignment deviation--where models fail to accurately localize salient text-referred regions in visual content. Moreover, without explicit guidance to model salient visual subjects, LMMs tend to over-rely on textual cues, resulting in visual modality neglect and suboptimal utilization of visual knowledge. To this end, we propose Salient Subject-Aware Multimodal Embedding (SSA-ME), a novel framework designed to enhance fine-grained representation learning through saliency-aware modeling. SSA-ME leverages LMMs and visual experts to identify and emphasize salient visual concepts in image-text pairs, and introduces a saliency-guided objective to better align cross-modal attention with semantically meaningful regions. Additionally, a feature regeneration module recalibrates visual features based on the derived saliency maps, ensuring a balanced and semantically coherent integration across modalities. Extensive experiments show that our method achieves state-of-the-art performance on the MMEB benchmark, demonstrating that incorporating subject-level modeling substantially improves multimodal retrieval. Comprehensive qualitative analyses further illustrate the interpretability and effectiveness of our approach.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 38 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

35 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs24 Apr
  2. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMs22 Dec
  3. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant2 Dec
  4. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2020Contrastive Learning with Hard Negative Samplescited by 628
  11. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  12. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
19 more
  1. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  2. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 176
  3. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  4. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  5. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  6. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
  7. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  8. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  9. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 58
  10. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
  11. 2024Interleaved-Modal Chain-of-Thoughtcited by 51
  12. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 49
  13. 2023GeneCIS: A Benchmark for General Conditional Image Similaritycited by 47
  14. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  15. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  16. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 22
  17. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 11
  18. 2023Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCEcited by 7
  19. 2023Gaussian Blur and Relative Edge Responsecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning-Augmented Representations for Multimodal Retrieval
  2. 2025Multimodal Representation Alignment for Cross-modal Information Retrieval
  3. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
  4. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
  5. 2025MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
  6. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  7. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond
  8. 2026Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning