mlsanityData to 7 Oct 2026

CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models

arXiv 2402.15021 ↗First posted 22 Feb 2024cs.CV cs.CLcited by 11

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent years have witnessed a significant increase in the performance of Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as CLIP, have been leveraged in multiple settings and demonstrated remarkable performance across several tasks. Such models excel at object-centric recognition yet learn text representations that seem invariant to word order, failing to compose known concepts in novel ways. However, no evidence exists that any VLM, including large-scale single-stream models such as GPT-4V, identifies compositions successfully. In this paper, we introduce a framework to significantly improve the ability of existing models to encode compositional language, with over 10% absolute improvement on compositionality benchmarks, while maintaining or improving the performance on standard object-recognition and retrieval benchmarks. Our code and pre-trained models are publicly available at https://github.com/netflix/clove.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 72 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

35 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  2. 2023Image Captioners Are Scalable Vision Learners Too13 Jun
  3. 2023COLA: A Benchmark for Compositional Text-to-image Retrieval5 May
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
18 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2017Localizing Moments in Video with Natural Languagecited by 660
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  9. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  11. 2022Patching open-vocabulary models by interpolating weightscited by 187
  12. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  13. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  14. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  15. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  16. 2016Captioning Images with Diverse Objectscited by 27
  17. 2020RareAct: A video dataset of unusual interactionscited by 23
  18. 2022FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Taskscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 17
  2. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  3. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  4. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  5. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  6. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Studycited by 4
  7. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognitioncited by 2
  8. 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Modelscited by 1
  9. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  10. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0
  11. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
  2. 2024In-Context Learning Improves Compositional Understanding of Vision-Language Models
  3. 2024ComAlign: Compositional Alignment in Vision-Language Models
  4. 2023Text encoders bottleneck compositionality in contrastive vision-language models
  5. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  6. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  7. 2026Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
  8. 2026Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality