mlsanityData to 7 Oct 2026

Variational Adapter for Cross-modal Similarity Representation

arXiv 2605.30968 ↗First posted 29 May 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The core of vision-language models lies in measuring cross-modal similarity within a unified representation space. However, most image-text matching or multi-class image classification datasets lack fine-grained cross-modal matching annotations, forcing the continuous similarity space into binary classification boundaries. This compression induces false negative samples and significantly impairs the generalization performance of cross-modal tasks. While prior research has attempted to mitigate this by modeling intra-modal ambiguity, it often overlooks inherent annotation flaws, leading to suboptimal uncertainty allocation. To address these challenges, we propose a Variational Adapter for Cross-modal Similarity Representation (VACSR). This approach reformulates image-text matching with fine-grained semantic scarcity as a variational inference problem. It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations. Experiments on image-text retrieval, domain generalization, and base-to-novel generalization demonstrate the proposed method's effectiveness and robust generalization ability.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 53 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

34 of 53 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  12. 2019Natural Adversarial Examplescited by 1,401
22 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  4. 2020Understanding the Behaviour of Contrastive Losscited by 397
  5. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  6. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  7. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  8. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  9. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 136
  10. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  11. 2023Consistency-guided Prompt Learning for Vision-Language Modelscited by 73
  12. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  13. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41
  14. 2023Improved Probabilistic Image-Text Representationscited by 40
  15. 2025MMRL: Multi-Modal Representation Learning for Vision-Language Modelscited by 38
  16. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  17. 2023BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistencycited by 27
  18. 2023Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrievalcited by 22
  19. 2021Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendationcited by 10
  20. 2023Negative Pre-aware for Noisy Cross-modal Matchingcited by 9
  21. 2023Dynamic Visual Semantic Sub-Embeddings and Fast Re-Rankingcited by 3
  22. 2025Towards Instance-wise Personalized Federated Learning via Semi-Implicit Bayesian Prompt Tuningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Trajectory-aware Cross-view Geo-localization with Sequential Observationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Rethinking Benchmarks for Cross-modal Image-text Retrieval
  2. 2024Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models
  3. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
  4. 2025Multimodal Representation Alignment for Cross-modal Information Retrieval
  5. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
  6. 2026CrossFlowDG: Bridging the Modality Gap with Cross-modal Flow Matching for Domain Generalization
  7. 2024Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval
  8. 2024Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models