mlsanityData to 14 Oct 2026

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

arXiv 2212.10537 ↗First posted 20 Dec 2022cs.CV cs.AI cs.CLcited by 102

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale neural network models combining text and images have made incredible progress in recent years. However, it remains an open question to what extent such models encode compositional representations of the concepts over which they operate, such as correctly identifying "red cube" by reasoning over the constituents "red" and "cube". In this work, we focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts and to bind variables in a structure-sensitive way (e.g., differentiating "cube behind sphere" from "sphere behind cube"). To inspect the performance of CLIP, we compare several architectures from research on compositional distributional semantics models (CDSMs), a line of research that attempts to implement traditional compositional linguistic structures within embedding spaces. We benchmark them on three synthetic datasets - single-object, two-object, and relational - designed to test concept binding. We find that CLIP can compose concepts in a single-object setting, but in situations where concept binding is needed, performance drops dramatically. At the same time, CDSMs also perform poorly, with best performance at chance level.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Builds on 42 of its 85 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

42 of 85 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  10. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  11. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
30 more
  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  2. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  3. 2015Neural Module Networkscited by 376
  4. 2015Skip-Thought Vectorscited by 322
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  6. 2019Good-Enough Compositional Data Augmentationcited by 158
  7. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  8. 2019Task-Driven Modular Networks for Zero-Shot Compositional Learningcited by 121
  9. 2021Learning Graph Embeddings for Compositional Zero-shot Learningcited by 112
  10. 2021Open World Compositional Zero-Shot Learningcited by 101
  11. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  12. 2022Meaning without reference in large language modelscited by 76
  13. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 67
  14. 2017Contrastive Learning for Image Captioningcited by 54
  15. 2019Distributional Semantics and Linguistic Theorycited by 53
  16. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  17. 2021Independent Prototype Propagation for Zero-Shot Compositionalitycited by 42
  18. 2020Compositional Video Synthesis with Action Graphscited by 34
  19. 2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32
  20. 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
  21. 2018RNNs Implicitly Implement Tensor Product Representationscited by 26
  22. 2019Compositional Generalization in Image Captioningcited by 23
  23. 2022Unit Testing for Concepts in Neural Networkscited by 23
  24. 2020Zero-Shot Learning with Common Sense Knowledge Graphscited by 17
  25. 2016Learning to generalize to new compositions in image understandingcited by 12
  26. 2021The Role of Syntactic Planning in Compositional Image Captioningcited by 12
  27. 2021Compositional Processing Emerges in Neural Networks Solving Math Problemscited by 11
  28. 2022Compositional Generalisation with Structured Reordering and Fertility Layerscited by 6
  29. 2016Functional Distributional Semanticscited by 3
  30. 2017Lexical and Derivational Meaning in Vector-Based Models of Relativisationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 30; 2025: 37; 2026: 24.

102 papers.

Sort:

By area:

  1. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  2. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  3. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  4. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  5. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
  6. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  7. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 31
  8. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understandingcited by 25
  9. 2023Prompting Language-Informed Distribution for Compositional Zero-Shot Learningcited by 24
  10. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Modelscited by 23
  11. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  12. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 22
38 more
  1. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21
  2. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 20
  3. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  4. 2024Pre-trained Vision-Language Models Learn Discoverable Visual Conceptscited by 16
  5. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  6. 2023Are Diffusion Models Vision-And-Language Reasoners?cited by 14
  7. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  8. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  9. 2024A Complexity-Based Theory of Compositionalitycited by 13
  10. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 13
  11. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language modelscited by 13
  12. 2024VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailmentcited by 10
  13. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
  14. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  15. 2024Towards Compositionality in Concept Learningcited by 7
  16. 2024Do Pre-trained Vision-Language Models Encode Object States?cited by 7
  17. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  18. 2025Scaling can lead to compositional generalizationcited by 7
  19. 2025Whither symbols in the era of advanced neural networks?cited by 7
  20. 2024Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Imagescited by 6
  21. 2024Finetuning CLIP to Reason about Pairwise Differencescited by 6
  22. 2024ResiDual Transformer Alignment with Spectral Decompositioncited by 6
  23. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  24. 2023SelfEval: Leveraging the discriminative nature of generative models for evaluationcited by 5
  25. 2024Improving Interpretability and Robustness for the Detection of AI-Generated Imagescited by 5
  26. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  27. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  28. 2025Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splattingcited by 5
  29. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  30. 2025Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Surveycited by 5
  31. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  32. 2025Evaluating Compositional Generalisation in VLMs and Diffusion Modelscited by 5
  33. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  34. 2024Improving Concept Alignment in Vision-Language Concept Bottleneck Modelscited by 4
  35. 2025Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Modelscited by 4
  36. 2025On the rankability of visual embeddingscited by 4
  37. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
  38. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3

50 most cited of 102 All 102 (JSON)