mlsanityData to 14 Oct 2026

Disentangling visual and written concepts in CLIP

arXiv 2206.07835 ↗First posted 15 Jun 2022cs.CVcited by 58

Not on the map. Why?

Abstract (arXiv)

The CLIP network measures the similarity between natural text and images; in this work, we investigate the entanglement of the representation of word images and natural images in its image encoder. First, we find that the image encoder has an ability to match word images with natural images of scenes described by those words. This is consistent with previous research that suggests that the meaning and the spelling of a word might be entangled deep within the network. On the other hand, we also find that CLIP has a strong ability to match nonsense words, suggesting that processing of letters is separated from processing of their meaning. To explicitly determine whether the spelling capability of CLIP is separable, we devise a procedure for identifying representation subspaces that selectively isolate or eliminate spelling capabilities. We benchmark our methods against a range of retrieval tasks, and we also test them by measuring the appearance of text in CLIP-guided generated images. We find that our methods are able to cleanly separate spelling capabilities of CLIP from the visual processing of natural images.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

17 of 26 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  9. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  10. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  11. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  12. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
5 more
  1. 2019On the "steerability" of generative adversarial networkscited by 248
  2. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  3. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  4. 2018Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networkscited by 129
  5. 2021Paint by Wordcited by 107

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 13; 2024: 17; 2025: 16; 2026: 7.

58 papers.

Sort:

By area:

  1. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236
  2. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  3. 2023Teaching CLIP to Count to Tencited by 198
  4. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  6. 2022Disentangled Representation Learningcited by 112
  7. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  8. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  9. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  10. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 45
  11. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 38
  12. 2023Defense-Prefix for Preventing Typographic Attacks on CLIPcited by 24
38 more
  1. 2023Zero-shot Model Diagnosiscited by 23
  2. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  3. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  4. 2023Parrot Captions Teach CLIP to Spot Textcited by 17
  5. 2025Hyperbolic Safety-Aware Vision-Language Modelscited by 17
  6. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalitiescited by 16
  7. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  8. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
  9. 2023Kiki or Bouba? Sound Symbolism in Vision-and-Language Modelscited by 14
  10. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 13
  11. 2023Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Modelscited by 12
  12. 2023Parts of Speech-Grounded Subspaces in Vision-Language Modelscited by 11
  13. 2025DIFFER: Disentangling Identity Features via Semantic Cues for Clothes-Changing Person Re-IDcited by 11
  14. 2025Towards an Explainable Comparison and Alignment of Feature Embeddingscited by 10
  15. 2024Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedbackcited by 9
  16. 2022Task Bias in Vision-Language Modelscited by 8
  17. 2025CLIPGaussian: Universal and Multimodal Style Transfer Based on Gaussian Splattingcited by 8
  18. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCRcited by 8
  19. 2024Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrievalcited by 6
  20. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
  21. 2023In-Style: Bridging Text and Uncurated Videos with Style Transfer for Text-Video Retrievalcited by 5
  22. 2024ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scalecited by 5
  23. 2025Transformation of audio embeddings into interpretable, concept-based representationscited by 5
  24. 2024Aligning Visual Contrastive learning models via Preference Optimizationcited by 4
  25. 2025Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentationcited by 4
  26. 2023SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspectivecited by 3
  27. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3
  28. 2024ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modelingcited by 2
  29. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  30. 2025Web Artifact Attacks Disrupt Vision Language Modelscited by 2
  31. 2025LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized Text-Guided Image Editingcited by 2
  32. 2025Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Modelscited by 2
  33. 2025Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensicscited by 1
  34. 2024Scene Depth Estimation from Traditional Oriental Landscape Paintingscited by 0
  35. 2024Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representationscited by 0
  36. 2024Attacks on multimodal modelscited by 0
  37. 2025Zero-Shot Visual Concept Blending Without Text Guidancecited by 0
  38. 2025Emergence of Text Readability in Vision Language Modelscited by 0

50 most cited of 58 All 58 (JSON)