mlsanityData to 14 Oct 2026

Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval

arXiv 2405.03190 ↗First posted 6 May 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In the recent years, the dual-encoder vision-language models (eg CLIP) have achieved remarkable text-to-image retrieval performance. However, we discover that these models usually results in very different retrievals for a pair of paraphrased queries. Such behavior might render the retrieval system less predictable and lead to user frustration. In this work, we consider the task of paraphrased text-to-image retrieval where a model aims to return similar results given a pair of paraphrased queries. To start with, we collect a dataset of paraphrased image descriptions to facilitate quantitative evaluation for this task. We then hypothesize that the undesired behavior of existing dual-encoder model is due to their text towers which are trained on image-sentence pairs and lack the ability to capture the semantic similarity between paraphrased queries. To improve on this, we investigate multiple strategies for training a dual-encoder model starting from a language model pretrained on a large text corpus. Compared to public dual-encoder models such as CLIP and OpenCLIP, the model trained with our best adaptation strategy achieves a significantly higher ranking similarity for paraphrased queries while maintaining similar zero-shot classification and retrieval accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 65 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

36 of 65 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
24 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  13. 2023Large Language Modelscited by 780
  14. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  15. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  16. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  17. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  18. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  19. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  20. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  21. 2017Query expansion techniques for information retrieval: A surveycited by 44
  22. 2017Measuring Personalization of Web Searchcited by 20
  23. 2022Rank List Sensitivity of Recommender Systems to Interaction Perturbationscited by 20
  24. 2023The Role of Pre-training Data in Transfer Learningcited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  2. 2025Adding simple structure at inference improves Vision-Language Compositionality
  3. 2025Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
  4. 2025Contrastive vision-language learning with paraphrasing and negation
  5. 2026Unified Multimodal and Multilingual Retrieval via Multi-Task Learning with NLU Integration
  6. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation
  7. 2025On the Brittleness of CLIP Text Encoders
  8. 2026Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models