# Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learning

arXiv: https://arxiv.org/abs/2402.05945 · HTML: https://mlsanity.com/paper/2402.05945/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Promises and Pitfalls of Black-Box Concept Learning Models](https://mlsanity.com/paper/2106.13314/) (2021)
- [Learning Concise and Descriptive Attributes for Visual Recognition](https://mlsanity.com/paper/2308.03685/) (2023)
- [Causality-based Neural Network Repair](https://mlsanity.com/paper/2204.09274/) (2022)
- [Provable Repair of Deep Neural Networks](https://mlsanity.com/paper/2104.04413/) (2021)
- [Explain Any Concept: Segment Anything Meets Concept-Based Explanation](https://mlsanity.com/paper/2305.10289/) (2023)

## Built on by (most cited first, 17 of 17)

- [VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance](https://mlsanity.com/paper/2408.01432/) (2024)
- [Improving Concept Alignment in Vision-Language Concept Bottleneck Models](https://mlsanity.com/paper/2405.01825/) (2024)
- [A Comprehensive Survey on the Risks and Limitations of Concept-based Models](https://mlsanity.com/paper/2506.04237/) (2025)
- [Incorporating Expert Rules into Neural Networks in the Framework of Concept-Based Learning](https://mlsanity.com/paper/2402.14726/) (2024)
- [Towards Reasonable Concept Bottleneck Models](https://mlsanity.com/paper/2506.05014/) (2025)
- [Rethinking Concept Bottleneck Models: From Pitfalls to Solutions](https://mlsanity.com/paper/2603.05629/) (2026)
- [FI-CBL: A Probabilistic Method for Concept-Based Learning with Expert Rules](https://mlsanity.com/paper/2406.19897/) (2024)
- [COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models](https://mlsanity.com/paper/2510.11012/) (2025)
- [Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?](https://mlsanity.com/paper/2606.22994/) (2026)
- [Improving ARDS Diagnosis Through Context-Aware Concept Bottleneck Models](https://mlsanity.com/paper/2508.09719/) (2025)
- [Exploring Concept Subspace for Self-explainable Text-Attributed Graph Learning](https://mlsanity.com/paper/2604.11986/) (2026)
- [Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models](https://mlsanity.com/paper/2606.04326/) (2026)
- [Formal Concept Lattices are Good Semantic Scaffolds for Concept-Based Learning](https://mlsanity.com/paper/2606.05471/) (2026)
- [In Defense of Information Leakage in Concept-based Models](https://mlsanity.com/paper/2606.10669/) (2026)
- [Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks](https://mlsanity.com/paper/2606.19489/) (2026)
- [TRACE: A Concept Bottleneck Model for Longitudinal 3D Glioblastoma Response Assessment](https://mlsanity.com/paper/2606.30313/) (2026)
- [On the Faithfulness of Post-Hoc Concept Bottleneck Models](https://mlsanity.com/paper/2606.30498/) (2026)

## Abstract (arXiv)

Concept Bottleneck Models (CBMs) aim to deliver interpretable and interventionable predictions by bridging features and labels with human-understandable concepts. While recent CBMs show promising potential, they suffer from information leakage, where unintended information beyond the concepts (either when concepts are represented with probabilities or binary states) are leaked to the subsequent label prediction. Consequently, distinct classes are falsely classified via indistinguishable concepts, undermining the interpretation and intervention of CBMs. This paper alleviates the information leakage issue by introducing label supervision in concept predication and constructing a hierarchical concept set. Accordingly, we propose a new paradigm of CBMs, namely SupCBM, which achieves label predication via predicted concepts and a deliberately-designed intervention matrix. SupCBM focuses on concepts that are mostly relevant to the predicted label and only distinguishes classes when different concepts are presented. Our evaluations show that SupCBM outperforms SOTA CBMs over diverse datasets. It also manifests better generality across different backbone models. With proper quantification of information leakage in different CBMs, we demonstrate that SupCBM significantly reduces the information leakage.