# Text encoders bottleneck compositionality in contrastive vision-language models

arXiv: https://arxiv.org/abs/2305.14897 · HTML: https://mlsanity.com/paper/2305.14897/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 24 May 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Captioning Images Taken by People Who Are Blind](https://mlsanity.com/paper/2002.08565/) (2020)
- [Benchmarking Spatial Relationships in Text-to-Image Generation](https://mlsanity.com/paper/2212.10015/) (2022)
- [Webly Supervised Concept Expansion for General Purpose Vision Models](https://mlsanity.com/paper/2202.02317/) (2022)
- [Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks](https://mlsanity.com/paper/2012.12352/) (2020)
- [Learning to count with deep object features](https://mlsanity.com/paper/1505.08082/) (2015)

## Built on by (most cited first, 30 of 34)

- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://mlsanity.com/paper/2406.13743/) (2024)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields](https://mlsanity.com/paper/2403.10997/) (2024)
- [INQUIRE: A Natural World Text-to-Image Retrieval Benchmark](https://mlsanity.com/paper/2411.02537/) (2024)
- [CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally](https://mlsanity.com/paper/2502.03566/) (2025)
- [A Philosophical Introduction to Language Models - Part II: The Way Forward](https://mlsanity.com/paper/2405.03207/) (2024)
- [VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models](https://mlsanity.com/paper/2504.03970/) (2025)
- [Do Audio-Visual Large Language Models Really See and Hear?](https://mlsanity.com/paper/2604.02605/) (2026)
- [Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning](https://mlsanity.com/paper/2505.19261/) (2025)
- [Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?](https://mlsanity.com/paper/2508.11011/) (2025)
- [LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval](https://mlsanity.com/paper/2601.14706/) (2026)
- [Causal Graphical Models for Vision-Language Compositional Understanding](https://mlsanity.com/paper/2412.09353/) (2024)
- [From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation](https://mlsanity.com/paper/2506.00633/) (2025)
- [SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation](https://mlsanity.com/paper/2603.22228/) (2026)
- [A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks](https://mlsanity.com/paper/2506.08227/) (2025)
- [Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization](https://mlsanity.com/paper/2509.25717/) (2025)
- [GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis](https://mlsanity.com/paper/2412.06089/) (2024)
- [Progressive Prompt Detailing for Improved Alignment in Text-to-Image Generative Models](https://mlsanity.com/paper/2503.17794/) (2025)
- [Multimodal Language Models as Text-to-Image Model Evaluators](https://mlsanity.com/paper/2505.00759/) (2025)
- [VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval](https://mlsanity.com/paper/2505.20291/) (2025)
- [SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models](https://mlsanity.com/paper/2511.12331/) (2025)
- [Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts](https://mlsanity.com/paper/2410.13030/) (2024)
- [Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions](https://mlsanity.com/paper/2510.16540/) (2025)
- [LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation](https://mlsanity.com/paper/2602.02220/) (2026)
- [LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?](https://mlsanity.com/paper/2605.11301/) (2026)
- [VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2404.16365/) (2024)
- [Enhancing CLIP Conceptual Embedding through Knowledge Distillation](https://mlsanity.com/paper/2412.03513/) (2024)
- [Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference](https://mlsanity.com/paper/2508.14280/) (2025)
- [GTMA: Dynamic Representation Optimization for OOD Vision-Language Models](https://mlsanity.com/paper/2512.18504/) (2025)

## Abstract (arXiv)

Performant vision-language (VL) models like CLIP represent captions using a single vector. How much information about language is lost in this bottleneck? We first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture (e.g., single object, to object+property, to multiple interacting objects). Then, we train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL models. This approach does not require images, allowing us to test on a broader range of scenes compared to prior work. We find that: 1) CLIP's text encoder falls short on more compositional inputs, including object relationships, attribute-object association, counting, and negations; 2) some text encoders work significantly better than others; and 3) text-only recovery performance predicts multi-modal matching performance on ControlledImCaps: a new evaluation benchmark we collect and release consisting of fine-grained compositional images and captions. Specifically, our results suggest text-only recoverability is a necessary (but not sufficient) condition for modeling compositional factors in contrastive VL models. We release our datasets and code.