mlsanityData to 14 Oct 2026

CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation

arXiv 2502.19842 ↗First posted 27 Feb 2025cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification, yet face challenges in complex multi-object scenarios. This study offers a comprehensive analysis of CLIP's limitations in these contexts using a specialized dataset, ComCO, designed to evaluate CLIP's encoders in diverse multi-object scenarios. Our findings reveal significant biases: the text encoder prioritizes first-mentioned objects, and the image encoder favors larger objects. Through retrieval and classification tasks, we quantify these biases across multiple CLIP variants and trace their origins to CLIP's training process, supported by analyses of the LAION dataset and training progression. Our image-text matching experiments show substantial performance drops when object size or token order changes, underscoring CLIP's instability with rephrased but semantically similar captions. Extending this to longer captions and text-to-image models like Stable Diffusion, we demonstrate how prompt order influences object prominence in generated images. For more details and access to our dataset and analysis code, visit our project repository: https://clip-oscope.github.io.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

21 of 26 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
9 more
  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  4. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  6. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  7. 2023Real-Time Flying Object Detection with YOLOv8cited by 94
  8. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  9. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioningcited by 7
  2. 2026BindCLIP: A Unified Contrastive-Generative Representation Learning Framework for Virtual Screeningcited by 2
  3. 2026Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Modelscited by 2
  4. 2025Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimizationcited by 1
  5. 2025GOATex: Geometry & Occlusion-Aware Texturingcited by 1
  6. 2026Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraintscited by 1
  7. 2026Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoningcited by 1
  8. 2026Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagerycited by 0
  9. 2026GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representationscited by 0
  10. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0
  11. 2026How Do Diffusion Classifiers Decide? A Bias-Centric Evaluationcited by 0
  12. 2026The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Modelcited by 0
2 more
  1. 2026A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagescited by 0
  2. 2026Anchor Divergence for Semantic Geometry in Contrastive Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  2. 2026CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
  3. 2024Toward a Holistic Evaluation of Robustness in CLIP Models
  4. 2024Diffusion Feedback Helps CLIP See Better
  5. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  6. 2025Object-centric Binding in Contrastive Language-Image Pretraining
  7. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  8. 2025Common Data Properties Limit Object-Attribute Binding in CLIP