# Promises and Pitfalls of Black-Box Concept Learning Models

arXiv: https://arxiv.org/abs/2106.13314 · HTML: https://mlsanity.com/paper/2106.13314/

## Key facts

- Cited by 131 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 12 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 8 of 8)

- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Do Concept Bottleneck Models Learn as Intended?](https://mlsanity.com/paper/2105.04289/) (2021)
- [Learning Latent Subspaces in Variational Autoencoders](https://mlsanity.com/paper/1812.06190/) (2018)
- [Interpretability Beyond Classification Output: Semantic Bottleneck Networks](https://mlsanity.com/paper/1907.10882/) (2019)
- [Semantic bottleneck for computer vision tasks](https://mlsanity.com/paper/1811.02234/) (2018)

## Built on by (most cited first, 30 of 131)

- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [Probabilistic Concept Bottleneck Models](https://mlsanity.com/paper/2306.01574/) (2023)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [GlanceNets: Interpretabile, Leak-proof Concept-based Models](https://mlsanity.com/paper/2205.15612/) (2022)
- [VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance](https://mlsanity.com/paper/2408.01432/) (2024)
- [A Closer Look at the Intervention Procedure of Concept Bottleneck Models](https://mlsanity.com/paper/2302.14260/) (2023)
- [Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off](https://mlsanity.com/paper/2209.09056/) (2022)
- [Concept Activation Regions: A Generalized Framework For Concept-Based Explanations](https://mlsanity.com/paper/2209.11222/) (2022)
- [Interpretable Neural-Symbolic Concept Reasoning](https://mlsanity.com/paper/2304.14068/) (2023)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Stochastic Concept Bottleneck Models](https://mlsanity.com/paper/2406.19272/) (2024)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Learning to Receive Help: Intervention-Aware Concept Embedding Models](https://mlsanity.com/paper/2309.16928/) (2023)
- [Human Uncertainty in Concept-Based AI Systems](https://mlsanity.com/paper/2303.12872/) (2023)
- [Auxiliary Losses for Learning Generalizable Concept-based Models](https://mlsanity.com/paper/2311.11108/) (2023)
- [Towards Robust Metrics for Concept Representation Evaluation](https://mlsanity.com/paper/2301.10367/) (2023)
- [Learning to Intervene on Concept Bottlenecks](https://mlsanity.com/paper/2308.13453/) (2023)
- [Causal Concept Graph Models: Beyond Causal Opacity in Deep Learning](https://mlsanity.com/paper/2405.16507/) (2024)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology](https://mlsanity.com/paper/2312.15010/) (2023)
- [Human-Centered Concept Explanations for Neural Networks](https://mlsanity.com/paper/2202.12451/) (2022)
- [Understanding and Enhancing Robustness of Concept-based Models](https://mlsanity.com/paper/2211.16080/) (2022)
- [Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning](https://mlsanity.com/paper/2309.07742/) (2023)
- [Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck](https://mlsanity.com/paper/2310.19660/) (2023)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)

## Abstract (arXiv)

Machine learning models that incorporate concept learning as an intermediate step in their decision making process can match the performance of black-box predictive models while retaining the ability to explain outcomes in human understandable terms. However, we demonstrate that the concept representations learned by these models encode information beyond the pre-defined concepts, and that natural mitigation strategies do not fully work, rendering the interpretation of the downstream prediction misleading. We describe the mechanism underlying the information leakage and suggest recourse for mitigating its effects.