mlsanityData to 14 Oct 2026

Linear Spaces of Meanings: Compositional Structures in Vision-Language Models

arXiv 2302.14383 ↗First posted 28 Feb 2023cs.LG cs.AI cs.CLcited by 48

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs). Traditionally, compositionality has been associated with algebraic operations on embeddings of words from a pre-existing vocabulary. In contrast, we seek to approximate representations from an encoder as combinations of a smaller set of vectors in the embedding space. These vectors can be seen as "ideal words" for generating concepts directly within the embedding space of the model. We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these compositional structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice. Finally, we empirically explore these structures in CLIP's embeddings and we evaluate their usefulness for solving different vision-language tasks such as classification, debiasing, and retrieval. Our results show that simple linear algebraic operations on embedding vectors can be used as compositional and interpretable methods for regulating the behavior of VLMs.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 50 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

28 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learning7 Apr

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  11. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  12. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
15 more
  1. 2018Towards a Definition of Disentangled Representationscited by 296
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  3. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
  4. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  5. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  6. 2019DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Imagescited by 134
  7. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  8. 2019Measuring Compositionality in Representation Learningcited by 84
  9. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  10. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 64
  11. 2018Towards Understanding Linear Word Analogiescited by 57
  12. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  13. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  14. 2021Learning Graph Embeddings for Open World Compositional Zero-Shot Learningcited by 49
  15. 2016Issues in evaluating semantic spaces using word analogiescited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 15; 2026: 12.

48 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  2. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  3. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  4. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  5. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  6. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  7. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  8. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  9. 2023TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generationcited by 18
  10. 2023Uncovering Meanings of Embeddings via Partial Orthogonalitycited by 15
  11. 2024A Complexity-Based Theory of Compositionalitycited by 13
  12. 2025Does Data Scaling Lead to Visual Compositional Generalization?cited by 13
36 more
  1. 2023Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Modelscited by 12
  2. 2025Four Principles for Physically Interpretable World Modelscited by 10
  3. 2025AI Agents as Universal Task Solverscited by 9
  4. 2025Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selectioncited by 8
  5. 2024Towards Compositionality in Concept Learningcited by 7
  6. 2026Learning Additively Compositional Latent Actions for Embodied AIcited by 7
  7. 2024Relation Rectification in Diffusion Modelcited by 6
  8. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Modelscited by 4
  9. 2025Concept Lancet: Image Editing with Compositional Representation Transplantcited by 4
  10. 2025On the rankability of visual embeddingscited by 4
  11. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
  12. 2024SHINE: Saliency-aware HIerarchical NEgative Ranking for Compositional Temporal Groundingcited by 3
  13. 2024LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registrationcited by 2
  14. 2024Contrastive Sparse Autoencoders for Interpreting Planning of Chess-Playing Agentscited by 2
  15. 2024Towards Utilising a Range of Neural Activations for Comprehending Representational Associationscited by 2
  16. 20257Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Modelscited by 2
  17. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  18. 2024Compositional Structures in Neural Embedding and Interaction Decompositionscited by 1
  19. 2024Simplifying complex machine learning by linearly separable network embedding spacescited by 1
  20. 2025Intervening to Learn and Compose Causally Disentangled Representationscited by 1
  21. 2025Mapping Semantic & Syntactic Relationships with Geometric Rotationcited by 1
  22. 2025Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Modelscited by 1
  23. 2025Learning by Analogy: A Causal Framework for Composition Generalizationcited by 1
  24. 2026Hierarchical Concept Embedding & Pursuit for Interpretable Image Classificationcited by 1
  25. 2026Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Modelscited by 1
  26. 2026NoisEasier: Test-Time Noise Optimization for Text-to-Video Generationcited by 1
  27. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  28. 2025Cross-modal Counterfactual Explanations: Uncovering Decision Factors and Dataset Biases in Subjective Classificationcited by 0
  29. 2026Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understandingcited by 0
  30. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  31. 2026Benchmarking Layout-Guided Diffusion Models through Unified Semantic-Spatial Evaluation in Closed and Open Settingscited by 0
  32. 2026How Far Does a Shared Linear Map Go? Probing Feature-Space Manipulability for Image Editingcited by 0
  33. 2026How can embedding models bind concepts?cited by 0
  34. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0
  35. 2026Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Groundingcited by 0
  36. 2026Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?
  3. 2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models
  4. 2023Compositional Semantics for Open Vocabulary Spatio-semantic Representations
  5. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  6. 2025Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
  7. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks
  8. 2024In-Context Learning Improves Compositional Understanding of Vision-Language Models