mlsanityData to 7 Oct 2026

Text encoders bottleneck compositionality in contrastive vision-language models

arXiv 2305.14897 ↗First posted 24 May 2023cs.CL cs.CV cs.LGcited by 34

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Performant vision-language (VL) models like CLIP represent captions using a single vector. How much information about language is lost in this bottleneck? We first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture (e.g., single object, to object+property, to multiple interacting objects). Then, we train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL models. This approach does not require images, allowing us to test on a broader range of scenes compared to prior work. We find that: 1) CLIP's text encoder falls short on more compositional inputs, including object relationships, attribute-object association, counting, and negations; 2) some text encoders work significantly better than others; and 3) text-only recovery performance predicts multi-modal matching performance on ControlledImCaps: a new evaluation benchmark we collect and release consisting of fine-grained compositional images and captions. Specifically, our results suggest text-only recoverability is a necessary (but not sufficient) condition for modeling compositional factors in contrastive VL models. We release our datasets and code.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 33 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

22 of 33 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
10 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  5. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  6. 2020Captioning Images Taken by People Who Are Blindcited by 108
  7. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  8. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  9. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  10. 2015Learning to count with deep object featurescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 14; 2026: 9.

34 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  3. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  4. 2024N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fieldscited by 34
  5. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  6. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  7. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
  8. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  9. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  10. 2025Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioningcited by 7
  11. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  12. 2025From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generationcited by 6
22 more
  1. 2025Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?cited by 6
  2. 2026LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrievalcited by 6
  3. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  4. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  5. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  6. 2025Progressive Prompt Detailing for Improved Alignment in Text-to-Image Generative Modelscited by 4
  7. 2026SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generationcited by 4
  8. 2025Multimodal Language Models as Text-to-Image Model Evaluatorscited by 3
  9. 2025VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrievalcited by 3
  10. 2025SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Modelscited by 3
  11. 2024Sensitivity of Generative VLMs to Semantically and Lexically Altered Promptscited by 2
  12. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  13. 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigationcited by 2
  14. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterationscited by 1
  15. 2024Enhancing CLIP Conceptual Embedding through Knowledge Distillationcited by 1
  16. 2025Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inferencecited by 1
  17. 2025GTMA: Dynamic Representation Optimization for OOD Vision-Language Modelscited by 0
  18. 2026Training-Free Text-to-Image Compositional Food Generation via Prompt Graftingcited by 0
  19. 2026LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?cited by 0
  20. 2026How can embedding models bind concepts?cited by 0
  21. 2026The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generationcited by 0
  22. 2026Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adding simple structure at inference improves Vision-Language Compositionality
  2. 2026A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
  3. 2024Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models
  4. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  5. 2026Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
  6. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
  7. 2024ComAlign: Compositional Alignment in Vision-Language Models
  8. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models