# Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts

arXiv: https://arxiv.org/abs/2408.02265 · HTML: https://mlsanity.com/paper/2408.02265/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Learning Concise and Descriptive Attributes for Visual Recognition](https://mlsanity.com/paper/2308.03685/) (2023)
- [Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions](https://mlsanity.com/paper/2401.02460/) (2024)
- [MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment](https://mlsanity.com/paper/2401.08527/) (2024)
- [Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis](https://mlsanity.com/paper/2404.05997/) (2024)

## Built on by (most cited first, 20 of 20)

- [Interpretable and Testable Vision Features via Sparse Autoencoders](https://mlsanity.com/paper/2502.06755/) (2025)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions](https://mlsanity.com/paper/2507.23784/) (2025)
- [Partially Shared Concept Bottleneck Models](https://mlsanity.com/paper/2511.22170/) (2025)
- [Zero-shot Concept Bottleneck Models](https://mlsanity.com/paper/2502.09018/) (2025)
- [A Comprehensive Survey on the Risks and Limitations of Concept-based Models](https://mlsanity.com/paper/2506.04237/) (2025)
- [A Geometric Unification of Concept Learning with Concept Cones](https://mlsanity.com/paper/2512.07355/) (2025)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [Flexible Concept Bottleneck Model](https://mlsanity.com/paper/2511.06678/) (2025)
- [Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning](https://mlsanity.com/paper/2601.23169/) (2026)
- [Caption Bottleneck Models](https://mlsanity.com/paper/2607.00578/) (2026)
- [ImageSet2Text: Describing Sets of Images through Text](https://mlsanity.com/paper/2503.19361/) (2025)
- [Chat-CBM: Towards Interactive Concept Bottleneck Models with Frozen Large Language Models](https://mlsanity.com/paper/2509.17522/) (2025)
- [Towards Safer and Understandable Driver Intention Prediction](https://mlsanity.com/paper/2510.09200/) (2025)
- [When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregate](https://mlsanity.com/paper/2512.03578/) (2025)
- [Concepts from Representations: Post-hoc Concept Bottleneck Models via Sparse Decomposition of Visual Representations](https://mlsanity.com/paper/2601.12303/) (2026)
- [Coverage-aware Semantic Representation Learning for Underrepresented Visual Concepts](https://mlsanity.com/paper/2602.16917/) (2026)
- [Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion](https://mlsanity.com/paper/2603.22624/) (2026)
- [Towards Fine-Grained and Verifiable Concept Bottleneck Models](https://mlsanity.com/paper/2605.14210/) (2026)
- [On the Faithfulness of Post-Hoc Concept Bottleneck Models](https://mlsanity.com/paper/2606.30498/) (2026)

## Abstract (arXiv)

The concept bottleneck model (CBM) is an interpretable-by-design framework that makes decisions by first predicting a set of interpretable concepts, and then predicting the class label based on the given concepts. Existing CBMs are trained with a fixed set of concepts (concepts are either annotated by the dataset or queried from language models). However, this closed-world assumption is unrealistic in practice, as users may wonder about the role of any desired concept in decision-making after the model is deployed. Inspired by the large success of recent vision-language pre-trained models such as CLIP in zero-shot classification, we propose "OpenCBM" to equip the CBM with open vocabulary concepts via: (1) Aligning the feature space of a trainable image feature extractor with that of a CLIP's image encoder via a prototype based feature alignment; (2) Simultaneously training an image classifier on the downstream dataset; (3) Reconstructing the trained classification head via any set of user-desired textual concepts encoded by CLIP's text encoder. To reveal potentially missing concepts from users, we further propose to iteratively find the closest concept embedding to the residual parameters during the reconstruction until the residual is small enough. To the best of our knowledge, our "OpenCBM" is the first CBM with concepts of open vocabularies, providing users the unique benefit such as removing, adding, or replacing any desired concept to explain the model's prediction even after a model is trained. Moreover, our model significantly outperforms the previous state-of-the-art CBM by 9% in the classification accuracy on the benchmark dataset CUB-200-2011.