mlsanityData to 7 Oct 2026

Interpreting CLIP's Image Representation via Text-Based Decomposition

arXiv 2310.05916 ↗First posted 9 Oct 2023cs.CV cs.AIcited by 189

Not on the map. Why?

Cited by 65 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the CLIP image encoder by analyzing how individual model components affect the final representation. We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands. Interpreting the attention heads, we characterize each head's role by automatically finding text representations that span its output space, which reveals property-specific roles for many heads (e.g. location or shape). Next, interpreting the image patches, we uncover an emergent spatial localization within CLIP. Finally, we use this understanding to remove spurious features from CLIP and to create a strong zero-shot image segmenter. Our results indicate that a scalable understanding of transformer models is attainable and can be used to repair and improve models.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

28 of 38 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
16 more
  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  4. 2023LERF: Language Embedded Radiance Fieldscited by 538
  5. 2021Image Segmentation Using Text and Image Promptscited by 509
  6. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  7. 2022Post-hoc Concept Bottleneck Modelscited by 289
  8. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  9. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  10. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  11. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  12. 2021Explaining in Style: Training a GAN to explain a classifier in StyleSpacecited by 122
  13. 2016Places: An Image Database for Deep Scene Understandingcited by 76
  14. 2022Disentangling visual and written concepts in CLIPcited by 57
  15. 2023Rosetta Neurons: Mining the Common Units in a Model Zoocited by 46
  16. 2021Deep Saliency Prior for Reducing Visual Distractioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 51; 2025: 83; 2026: 53.

189 papers.

Sort:

By area:

  1. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  2. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  5. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  6. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  7. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  8. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  9. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  10. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  11. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  12. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
38 more
  1. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 54
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  4. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 48
  5. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  6. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  7. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  8. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  9. 2024A Concept-Based Explainability Framework for Large Multimodal Modelscited by 39
  10. 2024Interpreting the Second-Order Effects of Neurons in CLIPcited by 36
  11. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  12. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  13. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  14. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
  15. 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 32
  16. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 28
  17. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  18. 2024Finding Visual Task Vectorscited by 24
  19. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  20. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMscited by 24
  21. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  22. 2025Multimodal Language Models See Better When They Look Shallowercited by 22
  23. 2024Interpreting Object-level Foundation Models via Visual Precision Searchcited by 20
  24. 2025Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Videocited by 20
  25. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  26. 2024Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIPcited by 19
  27. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18
  28. 2024Robust AI-Generated Text Detection by Restricted Embeddingscited by 18
  29. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 16
  30. 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 15
  31. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
  32. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  33. 2025Visual Representations inside the Language Modelcited by 13
  34. 2025Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Modelscited by 12
  35. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Setcited by 12
  36. 2023Interpreting and Controlling Vision Foundation Models via Text Explanationscited by 11
  37. 2024Feature contamination: Neural networks learn uncorrelated features and fail to generalizecited by 11
  38. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11

50 most cited of 189 All 189 (JSON)