mlsanityData to 7 Oct 2026

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

arXiv 2406.09388 ↗First posted 13 Jun 2024cs.CV cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Vision and language models (VLMs) such as CLIP have showcased remarkable zero-shot recognition abilities yet face challenges in visio-linguistic compositionality, particularly in linguistic comprehension and fine-grained image-text alignment. This paper explores the intricate relationship between compositionality and recognition -- two pivotal aspects of VLM capability. We conduct a comprehensive evaluation of existing VLMs, covering both pre-training approaches aimed at recognition and the fine-tuning methods designed to improve compositionality. Our evaluation employs 12 benchmarks for compositionality, along with 21 zero-shot classification and two retrieval benchmarks for recognition. In our analysis from 274 CLIP model checkpoints, we reveal patterns and trade-offs that emerge between compositional understanding and recognition accuracy. Ultimately, this necessitates strategic efforts towards developing models that improve both capabilities, as well as the meticulous formulation of benchmarks for compositionality. We open our evaluation framework at https://github.com/ytaek-oh/vl_compo.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

36 of 42 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  7. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  10. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  11. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  12. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
24 more
  1. 2023Improving CLIP Training with Language Rewritescited by 216
  2. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  4. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  5. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  6. 2023Hyperbolic Image-Text Representationscited by 132
  7. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  8. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  9. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  10. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  11. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  12. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  13. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  14. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  15. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  16. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  17. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  18. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  19. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36
  20. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  21. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  22. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  23. 2024ColorSwap: A Color and Word Order Dataset for Multimodal Evaluationcited by 11
  24. 2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Modelscited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  2. 2025When 'YES' Meets 'BUT': Can Large Models Comprehend Contradictory Humor Through Comparative Reasoning?cited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  2. 2024ComAlign: Compositional Alignment in Vision-Language Models
  3. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
  4. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  5. 2026Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
  6. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
  7. 2026Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
  8. 2024In-Context Learning Improves Compositional Understanding of Vision-Language Models