mlsanityData to 7 Oct 2026

Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)

arXiv 2402.10376 ↗First posted 16 Feb 2024cs.LG cs.CVcited by 116

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

CLIP embeddings have demonstrated remarkable performance across a wide range of multimodal applications. However, these high-dimensional, dense vector representations are not easily interpretable, limiting our understanding of the rich structure of CLIP and its use in downstream applications that require transparency. In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representations into semantic concepts. We formulate this problem as one of sparse recovery and propose a novel method, Sparse Linear Concept Embeddings, for transforming CLIP representations into sparse linear combinations of human-interpretable concepts. Distinct from previous work, SpLiCE is task-agnostic and can be used, without training, to explain and even replace traditional dense CLIP representations, maintaining high downstream performance while significantly improving their interpretability. We also demonstrate significant use cases of SpLiCE representations including detecting spurious correlations and model editing.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Builds on 39 of its 78 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

39 of 78 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2020Concept Bottleneck Modelscited by 1,022
  11. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  12. 2016The Mythos of Model Interpretabilitycited by 716
27 more
  1. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  6. 2022Post-hoc Concept Bottleneck Modelscited by 289
  7. 2023Label-Free Concept Bottleneck Modelscited by 282
  8. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  9. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  10. 2022Linearly Mapping from Image to Text Spacecited by 139
  11. 2018Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networkscited by 128
  12. 2021Promises and Pitfalls of Black-Box Concept Learning Modelscited by 128
  13. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  14. 2021Do Concept Bottleneck Models Learn as Intended?cited by 114
  15. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  16. 2013Deep Learning of Representations: Looking Forwardcited by 84
  17. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  18. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  19. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  20. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 63
  21. 2015Sparse Overcomplete Word Vector Representationscited by 43
  22. 2023Disentanglement via Latent Quantizationcited by 40
  23. 2020On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectorscited by 39
  24. 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
  25. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  26. 2023STAIR: Learning Sparse Text and Image Representation in Grounded Tokenscited by 23
  27. 2023Sparse Linear Concept Discovery Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 52; 2026: 49.

116 papers.

Sort:

By area:

  1. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  2. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  3. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  4. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  5. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 54
  6. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  7. 2024A Concept-Based Explainability Framework for Large Multimodal Modelscited by 39
  8. 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
  9. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  10. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  11. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 28
  12. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
38 more
  1. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Setcited by 12
  2. 2024Crafting Interpretable Embeddings by Asking LLMs Questionscited by 11
  3. 2025Object Centric Concept Bottleneckscited by 10
  4. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 9
  5. 2024Unified Lexical Representation for Interpretable Visual-Language Alignmentcited by 8
  6. 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Modelscited by 7
  7. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  8. 2025Rethinking Explainability in the Era of Multimodal AIcited by 6
  9. 2025Quantifying Structure in CLIP Embeddings: A Statistical Framework for Concept Interpretationcited by 6
  10. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  11. 2025Enhancing Concept Localization in CLIP-based Concept Bottleneck Modelscited by 6
  12. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoderscited by 6
  13. 2025Transformation of audio embeddings into interpretable, concept-based representationscited by 5
  14. 2025ConceptScope: Characterizing Dataset Bias via Disentangled Visual Conceptscited by 5
  15. 2025Partially Shared Concept Bottleneck Modelscited by 5
  16. 2024ResiDual Transformer Alignment with Spectral Decompositioncited by 4
  17. 2025Zero-shot Concept Bottleneck Modelscited by 4
  18. 2025From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliancecited by 4
  19. 2025SUA: Stealthy Multimodal Large Language Model Unlearning Attackcited by 4
  20. 2024I Bet You Did Not Mean That: Testing Semantic Importance via Bettingcited by 3
  21. 2024DDR: Exploiting Deep Degradation Response as Flexible Image Descriptorcited by 3
  22. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3
  23. 2025All-in-One Slider for Attribute Manipulation in Diffusion Modelscited by 3
  24. 2025Model Science: getting serious about verification, explanation and control of AI systemscited by 3
  25. 2025Towards Interpretable Geo-localization: a Concept-Aware Global Image-GPS Alignment Frameworkcited by 3
  26. 2025ProtSAE: Disentangling and Interpreting Protein Language Models via Semantically-Guided Sparse Autoencoderscited by 3
  27. 2025SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Modelscited by 3
  28. 2025A Geometric Unification of Concept Learning with Concept Conescited by 3
  29. 2026Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformerscited by 3
  30. 2024ComFe: An Interpretable Head for Vision Transformerscited by 2
  31. 2024Operationalizing the Blueprint for an AI Bill of Rights: Recommendations for Practitioners, Researchers, and Policy Makerscited by 2
  32. 2025Coreset Selection via LLM-based Concept Bottleneckscited by 2
  33. 2025The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Modelscited by 2
  34. 2025Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detectioncited by 2
  35. 2025MedSAE: Dissecting MedCLIP Representations with Sparse Autoencoderscited by 2
  36. 2025Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approachcited by 2
  37. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  38. 2026Decomposing multimodal embedding spaces with group-sparse autoencoderscited by 2

50 most cited of 116 All 116 (JSON)