# Common Data Properties Limit Object-Attribute Binding in CLIP

arXiv: https://arxiv.org/abs/2507.07985 · HTML: https://mlsanity.com/paper/2507.07985/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models](https://mlsanity.com/paper/2305.19595/) (2023)
- [Teaching Structured Vision&Language Concepts to Vision&Language Models](https://mlsanity.com/paper/2211.11733/) (2022)
- [TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives](https://mlsanity.com/paper/2411.02545/) (2024)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning](https://mlsanity.com/paper/1809.10780/) (2018)
- [Linear Spaces of Meanings: Compositional Structures in Vision-Language Models](https://mlsanity.com/paper/2302.14383/) (2023)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning](https://mlsanity.com/paper/2308.03977/) (2023)
- [CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally](https://mlsanity.com/paper/2502.03566/) (2025)
- [When are Lemons Purple? The Concept Association Bias of Vision-Language Models](https://mlsanity.com/paper/2212.12043/) (2022)
- [Object-centric Binding in Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2502.14113/) (2025)

## Built on by (most cited first, 4 of 4)

- [SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery](https://mlsanity.com/paper/2602.17395/) (2026)
- [No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models](https://mlsanity.com/paper/2603.25722/) (2026)
- [How can embedding models bind concepts?](https://mlsanity.com/paper/2605.31503/) (2026)
- [BindCLIP: One Balanced Coupling For Compositional Vision Language Scoring](https://mlsanity.com/paper/2609.23717/) (2026)

## Abstract (arXiv)

Contrastive vision-language models like CLIP are used for a large variety of applications, such as zero-shot classification or as vision encoder for multi-modal models. Despite their popularity, their representations show major limitations. For instance, CLIP models learn bag-of-words representations and, as a consequence, fail to distinguish whether an image is of ``a yellow submarine and a blue bus'' or ``a blue submarine and a yellow bus''. Previous attempts to fix this issue added hard negatives during training or modified the architecture, but failed to resolve the problem in its entirety. We suspect that the missing insights to solve the binding problem for CLIP are hidden in arguably the most important part of learning algorithms: the data. In this work, we fill this gap by rigorously identifying the influence of data properties on CLIP's ability to learn binding using a synthetic dataset. We find that common properties of natural data such as low attribute density, incomplete captions, and the saliency bias, a tendency of human captioners to describe the object that is ``most salient'' to them, have a detrimental effect on binding performance. In contrast to common belief, we find that neither scaling the batch size, i.e., implicitly adding more hard negatives, nor explicitly creating hard negatives enables CLIP to learn reliable binding. Only when the data expresses our identified data properties does CLIP learn almost perfect binding.