mlsanityData to 7 Oct 2026

Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge

arXiv 2410.13016 ↗First posted 16 Oct 2024cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This work provides a new approach for interpreting CLIP models for image classification from the lens of mutual knowledge between the two modalities. Specifically, we ask: what concepts do both vision and language CLIP encoders learn in common that influence the joint embedding space, causing points to be closer or further apart? We answer this question via an approach of textual concept-based explanations, showing their effectiveness, and perform an analysis encompassing a pool of 13 CLIP models varying in architecture, size and pretraining datasets. We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions. Our approach demonstrates an effective and human-friendly way of understanding zero-shot classification decisions with CLIP.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

49 of 64 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,350
37 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2020Concept Bottleneck Modelscited by 1,022
  7. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  8. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  13. 2022Visual Classification via Description from Large Language Modelscited by 331
  14. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  15. 2023Label-Free Concept Bottleneck Modelscited by 282
  16. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  17. 2018DropBlock: A regularization method for convolutional networkscited by 261
  18. 2023Data Filtering Networkscited by 247
  19. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  20. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  21. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  22. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  23. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  24. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  25. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  26. 2024FeatUp: A Model-Agnostic Framework for Features at Any Resolutioncited by 114
  27. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  28. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 100
  29. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  30. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  31. 2023Fine-Grained Visual Promptingcited by 81
  32. 2018Deep Feature Factorization For Concept Discoverycited by 64
  33. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  34. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  35. 2023Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Taskscited by 12
  36. 2023DeViL: Decoding Vision features into Languagecited by 10
  37. 2023Prototype-based Dataset Comparisoncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  2. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  3. 2025Align 3D Representation and Text Embedding for 3D Content Personalizationcited by 2
  4. 2025Principled Coarse-Grained Acceptance for Speculative Decoding in Speechcited by 2
  5. 2026GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learningcited by 2
  6. 2025Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detectioncited by 1
  7. 2025EvoVLMA: Evolutionary Vision-Language Model Adaptationcited by 1
  8. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  9. 2025SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarkingcited by 0
  10. 2026DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interactioncited by 0
  11. 2026SAS: Semantic-aware Sampling for Generative Dataset Distillationcited by 0
  12. 2026Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Modelcited by 0