# CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation

arXiv: https://arxiv.org/abs/2502.19842 · HTML: https://mlsanity.com/paper/2502.19842/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations](https://mlsanity.com/paper/2207.00221/) (2022)
- [Real-Time Flying Object Detection with YOLOv8](https://mlsanity.com/paper/2305.09972/) (2023)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining](https://mlsanity.com/paper/2311.03964/) (2023)

## Built on by (most cited first, 14 of 14)

- [Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning](https://mlsanity.com/paper/2505.19261/) (2025)
- [BindCLIP: A Unified Contrastive-Generative Representation Learning Framework for Virtual Screening](https://mlsanity.com/paper/2602.15236/) (2026)
- [Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2603.22042/) (2026)
- [Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization](https://mlsanity.com/paper/2503.07038/) (2025)
- [GOATex: Geometry & Occlusion-Aware Texturing](https://mlsanity.com/paper/2511.23051/) (2025)
- [Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraints](https://mlsanity.com/paper/2601.14207/) (2026)
- [Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning](https://mlsanity.com/paper/2604.21346/) (2026)
- [Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagery](https://mlsanity.com/paper/2601.21159/) (2026)
- [GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations](https://mlsanity.com/paper/2605.17727/) (2026)
- [Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs](https://mlsanity.com/paper/2606.01710/) (2026)
- [How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation](https://mlsanity.com/paper/2607.03831/) (2026)
- [The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model](https://mlsanity.com/paper/2607.13660/) (2026)
- [A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2610.04300/) (2026)
- [Anchor Divergence for Semantic Geometry in Contrastive Learning](https://mlsanity.com/paper/2610.06919/) (2026)

## Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification, yet face challenges in complex multi-object scenarios. This study offers a comprehensive analysis of CLIP's limitations in these contexts using a specialized dataset, ComCO, designed to evaluate CLIP's encoders in diverse multi-object scenarios. Our findings reveal significant biases: the text encoder prioritizes first-mentioned objects, and the image encoder favors larger objects. Through retrieval and classification tasks, we quantify these biases across multiple CLIP variants and trace their origins to CLIP's training process, supported by analyses of the LAION dataset and training progression. Our image-text matching experiments show substantial performance drops when object size or token order changes, underscoring CLIP's instability with rephrased but semantically similar captions. Extending this to longer captions and text-to-image models like Stable Diffusion, we demonstrate how prompt order influences object prominence in generated images. For more details and access to our dataset and analysis code, visit our project repository: https://clip-oscope.github.io.