# CLIP-QDA: An Explainable Concept Bottleneck Model

arXiv: https://arxiv.org/abs/2312.00110 · HTML: https://mlsanity.com/paper/2312.00110/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests](https://mlsanity.com/paper/1509.02237/) (2015)
- [A Deeper Look at Dataset Bias](https://mlsanity.com/paper/1505.01257/) (2015)
- [Model Agnostic Supervised Local Explanations](https://mlsanity.com/paper/1807.02910/) (2018)
- [Learning Concise and Descriptive Attributes for Visual Recognition](https://mlsanity.com/paper/2308.03685/) (2023)
- [Very Deep Transformers for Neural Machine Translation](https://mlsanity.com/paper/2008.07772/) (2020)
- [Interpretability Beyond Classification Output: Semantic Bottleneck Networks](https://mlsanity.com/paper/1907.10882/) (2019)
- [Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models](https://mlsanity.com/paper/2310.03182/) (2023)
- [Finding and Fixing Spurious Patterns with Explanations](https://mlsanity.com/paper/2106.02112/) (2021)

## Built on by (most cited first, 13 of 13)

- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [Benchmarking XAI Explanations with Human-Aligned Evaluations](https://mlsanity.com/paper/2411.02470/) (2024)
- [Enhancing Concept Localization in CLIP-based Concept Bottleneck Models](https://mlsanity.com/paper/2510.07115/) (2025)
- [Incorporating Expert Rules into Neural Networks in the Framework of Concept-Based Learning](https://mlsanity.com/paper/2402.14726/) (2024)
- [Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions](https://mlsanity.com/paper/2408.14153/) (2024)
- [A Geometric Unification of Concept Learning with Concept Cones](https://mlsanity.com/paper/2512.07355/) (2025)
- [On the Explainability of Vision-Language Models in Art History](https://mlsanity.com/paper/2602.20853/) (2026)
- [FI-CBL: A Probabilistic Method for Concept-Based Learning with Expert Rules](https://mlsanity.com/paper/2406.19897/) (2024)
- [Concept-Based Mechanistic Interpretability Using Structured Knowledge Graphs](https://mlsanity.com/paper/2507.05810/) (2025)
- [Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency](https://mlsanity.com/paper/2406.08840/) (2024)
- [Beyond Semantics: Modeling Factual and Affective Perceptual Experiences from Vision-Language Data](https://mlsanity.com/paper/2606.03345/) (2026)
- [On the Faithfulness of Post-Hoc Concept Bottleneck Models](https://mlsanity.com/paper/2606.30498/) (2026)
- [CHOQOLATE: Organizing Concept Bottleneck Latent Spaces with Choquet Integrals](https://mlsanity.com/paper/2609.37786/) (2026)

## Abstract (arXiv)

In this paper, we introduce an explainable algorithm designed from a multi-modal foundation model, that performs fast and explainable image classification. Drawing inspiration from CLIP-based Concept Bottleneck Models (CBMs), our method creates a latent space where each neuron is linked to a specific word. Observing that this latent space can be modeled with simple distributions, we use a Mixture of Gaussians (MoG) formalism to enhance the interpretability of this latent space. Then, we introduce CLIP-QDA, a classifier that only uses statistical values to infer labels from the concepts. In addition, this formalism allows for both local and global explanations. These explanations come from the inner design of our architecture, our work is part of a new family of greybox models, combining performances of opaque foundation models and the interpretability of transparent models. Our empirical findings show that in instances where the MoG assumption holds, CLIP-QDA achieves similar accuracy with state-of-the-art methods CBMs. Our explanations compete with existing XAI methods while being faster to compute.