# CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally

arXiv: https://arxiv.org/abs/2502.03566 · HTML: https://mlsanity.com/paper/2502.03566/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 43 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Visual Spatial Reasoning](https://mlsanity.com/paper/2205.00363/) (2022)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations](https://mlsanity.com/paper/2207.00221/) (2022)
- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [The Neglected Tails in Vision-Language Models](https://mlsanity.com/paper/2401.12425/) (2024)
- [TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives](https://mlsanity.com/paper/2411.02545/) (2024)
- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [Mass-Producing Failures of Multimodal Systems with Language Models](https://mlsanity.com/paper/2306.12105/) (2023)
- [Text encoders bottleneck compositionality in contrastive vision-language models](https://mlsanity.com/paper/2305.14897/) (2023)
- [Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?](https://mlsanity.com/paper/2203.17271/) (2022)
- [PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning](https://mlsanity.com/paper/2308.03977/) (2023)
- [Prompting Language-Informed Distribution for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2305.14428/) (2023)

## Built on by (most cited first, 30 of 31)

- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [Does Data Scaling Lead to Visual Compositional Generalization?](https://mlsanity.com/paper/2507.07102/) (2025)
- [Diffusion Classifiers Understand Compositionality, but Conditions Apply](https://mlsanity.com/paper/2505.17955/) (2025)
- [Noise Matters: Optimizing Matching Noise for Diffusion Classifiers](https://mlsanity.com/paper/2508.11330/) (2025)
- [A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks](https://mlsanity.com/paper/2506.08227/) (2025)
- [Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning](https://mlsanity.com/paper/2602.21035/) (2026)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [Common Data Properties Limit Object-Attribute Binding in CLIP](https://mlsanity.com/paper/2507.07985/) (2025)
- [VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval](https://mlsanity.com/paper/2505.20291/) (2025)
- [Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions](https://mlsanity.com/paper/2604.17358/) (2026)
- [DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models](https://mlsanity.com/paper/2510.15015/) (2025)
- [RLM: A Vision-Language Model Approach for Radar Scene Understanding](https://mlsanity.com/paper/2511.21105/) (2025)
- [ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching](https://mlsanity.com/paper/2512.17178/) (2025)
- [When Negation Is a Geometry Problem in Vision-Language Models](https://mlsanity.com/paper/2603.20554/) (2026)
- [Latent Goal Prediction from Language for Model-Based Planning](https://mlsanity.com/paper/2606.20627/) (2026)
- [Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models](https://mlsanity.com/paper/2607.23052/) (2026)
- [Evaluating Attribute Confusion in Fashion Text-to-Image Generation](https://mlsanity.com/paper/2507.07079/) (2025)
- [Perceptual Reality Transformer: What Must an Illustration Preserve?](https://mlsanity.com/paper/2508.09852/) (2025)
- [Half-Truths Break Similarity-Based Retrieval](https://mlsanity.com/paper/2602.23906/) (2026)
- [Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Family](https://mlsanity.com/paper/2604.05971/) (2026)
- [Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding](https://mlsanity.com/paper/2604.13313/) (2026)
- [OccDirector: Language-Guided Behavior and Interaction Generation in 4D Occupancy Space](https://mlsanity.com/paper/2604.22240/) (2026)
- [How can embedding models bind concepts?](https://mlsanity.com/paper/2605.31503/) (2026)
- [Formalizing the Binding Problem](https://mlsanity.com/paper/2606.03976/) (2026)
- [Foveated Probes Recover Localized Binding Information in Vision Foundation Models](https://mlsanity.com/paper/2608.00726/) (2026)
- [HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding](https://mlsanity.com/paper/2608.29120/) (2026)
- [FRAME: Factored Retrieval via Attribute Readouts for Object-Centric Scene Memory](https://mlsanity.com/paper/2609.08886/) (2026)
- [BindCLIP: One Balanced Coupling For Compositional Vision Language Scoring](https://mlsanity.com/paper/2609.23717/) (2026)
- [Don't CLAP: Are Music-Text Models Bag-of-Words?](https://mlsanity.com/paper/2609.30540/) (2026)
- [Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry](https://mlsanity.com/paper/2609.37230/) (2026)

## Abstract (arXiv)

CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks. However, recent studies have questioned its ability to represent compositional concepts effectively. These works suggest that CLIP often acts like a bag-of-words (BoW) model, interpreting images and text as sets of individual concepts without grasping the structural relationships. In particular, CLIP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text. In this work, we investigate why CLIP exhibits this BoW-like behavior. Our key finding is that CLIP does not lack binding information. Through linear probing, robustness tests with increasing object counts, and conjunctive search experiments, we show that attribute-object bindings are already encoded within CLIP's text and image embeddings. The weakness lies in the cross-modal alignment, which fails to preserve this information. We show it can be accessed cross-modally with a simple linear transformation to text embeddings. This improves CLIP's attribute-object binding performance and confirms that the information was already encoded unimodally. In practice, this means CLIP-based systems can be enhanced with a lightweight linear layer trained on existing embeddings, avoiding costly encoder retraining. The code is available at https://github.com/kdariina/CLIP-not-BoW-unimodally.