mlsanityData to 14 Oct 2026

Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs

arXiv 2509.19207 ↗First posted 23 Sep 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive vision-language models (VLMs) have made significant progress in binding visual and textual information, yet understanding long, compositional captions remains an open challenge. While these capabilities are often assumed to be closely related, the conditions under which they reinforce each other remain unclear. In this paper, we empirically analyze when compositional reasoning and long-caption understanding transfer across tasks, and when this relationship fails. Through controlled experiments across diverse training objectives, datasets, and architectural designs, we find a bidirectional but sensitive relationship between the two capabilities. Models trained on poorly grounded captions or with limited parameter updates fail to generalize, while high-quality long-caption data with strong visual grounding promotes both capabilities simultaneously. We further show that architectural choices aimed at preserving general alignment, such as frozen positional embeddings, can inadvertently limit compositional learning. Our analysis provides actionable guidelines for data selection and model design to improve VLM generalization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 48 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

30 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  2. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding15 Jun
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  4. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
14 more
  1. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  2. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  3. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  5. 2019Connecting Vision and Language with Localized Narrativescited by 226
  6. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  7. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  8. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  9. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  10. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  11. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
  12. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  13. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  14. 2019Compositional Generalization in Image Captioningcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions
  2. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  3. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
  4. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
  5. 2024In-Context Learning Improves Compositional Understanding of Vision-Language Models
  6. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks
  7. 2024ComAlign: Compositional Alignment in Vision-Language Models
  8. 2023Text encoders bottleneck compositionality in contrastive vision-language models