# Label-Free Concept Bottleneck Models

arXiv: https://arxiv.org/abs/2304.06129 · HTML: https://mlsanity.com/paper/2304.06129/

## Key facts

- Cited by 285 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 12 Apr 2023.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [Rewriting a Deep Generative Model](https://mlsanity.com/paper/2007.15646/) (2020)
- [Leveraging Sparse Linear Layers for Debuggable Deep Networks](https://mlsanity.com/paper/2105.04857/) (2021)
- [Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off](https://mlsanity.com/paper/2209.09056/) (2022)
- [Interpretability Beyond Classification Output: Semantic Bottleneck Networks](https://mlsanity.com/paper/1907.10882/) (2019)
- [NBDT: Neural-Backed Decision Trees](https://mlsanity.com/paper/2004.00221/) (2020)
- [Rewriting Geometric Rules of a GAN](https://mlsanity.com/paper/2207.14288/) (2022)

## Built on by (most cited first, 30 of 285)

- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance](https://mlsanity.com/paper/2408.01432/) (2024)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [Stochastic Concept Bottleneck Models](https://mlsanity.com/paper/2406.19272/) (2024)
- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [Concept Bottleneck Large Language Models](https://mlsanity.com/paper/2412.07992/) (2024)
- [Learning to Receive Help: Intervention-Aware Concept Embedding Models](https://mlsanity.com/paper/2309.16928/) (2023)
- [Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models](https://mlsanity.com/paper/2310.03182/) (2023)
- [Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography](https://mlsanity.com/paper/2405.12255/) (2024)
- [Steering CLIP's vision transformer with sparse autoencoders](https://mlsanity.com/paper/2504.08729/) (2025)
- [Auxiliary Losses for Learning Generalizable Concept-based Models](https://mlsanity.com/paper/2311.11108/) (2023)
- [Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations](https://mlsanity.com/paper/2401.14142/) (2024)
- [Coarse-to-Fine Concept Bottleneck Models](https://mlsanity.com/paper/2310.02116/) (2023)
- [Learning to Intervene on Concept Bottlenecks](https://mlsanity.com/paper/2308.13453/) (2023)
- [Causal Concept Graph Models: Beyond Causal Opacity in Deep Learning](https://mlsanity.com/paper/2405.16507/) (2024)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology](https://mlsanity.com/paper/2503.14911/) (2025)
- [Language Guided Concept Bottleneck Models for Interpretable Continual Learning](https://mlsanity.com/paper/2503.23283/) (2025)
- [Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework](https://mlsanity.com/paper/2403.07636/) (2024)
- [SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology](https://mlsanity.com/paper/2312.15010/) (2023)
- [Editable Concept Bottleneck Models](https://mlsanity.com/paper/2405.15476/) (2024)
- [Sparse Linear Concept Discovery Models](https://mlsanity.com/paper/2308.10782/) (2023)
- [Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning](https://mlsanity.com/paper/2309.07742/) (2023)
- [Concept Bottleneck with Visual Concept Filtering for Explainable Medical Image Classification](https://mlsanity.com/paper/2308.11920/) (2023)
- [Semi-supervised Concept Bottleneck Models](https://mlsanity.com/paper/2406.18992/) (2024)
- [Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts](https://mlsanity.com/paper/2408.02265/) (2024)
- [A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts](https://mlsanity.com/paper/2406.10368/) (2024)

## Abstract (arXiv)

Concept bottleneck models (CBM) are a popular way of creating more interpretable neural networks by having hidden layer neurons correspond to human-understandable concepts. However, existing CBMs and their variants have two crucial limitations: first, they need to collect labeled data for each of the predefined concepts, which is time consuming and labor intensive; second, the accuracy of a CBM is often significantly lower than that of a standard neural network, especially on more complex datasets. This poor performance creates a barrier for adopting CBMs in practical real world applications. Motivated by these challenges, we propose Label-free CBM which is a novel framework to transform any neural network into an interpretable CBM without labeled concept data, while retaining a high accuracy. Our Label-free CBM has many advantages, it is: scalable - we present the first CBM scaled to ImageNet, efficient - creating a CBM takes only a few hours even for very large datasets, and automated - training it for a new dataset requires minimal human effort. Our code is available at https://github.com/Trustworthy-ML-Lab/Label-free-CBM. Finally, in Appendix B we conduct a large scale user evaluation of the interpretability of our method.