# Interpretability Beyond Classification Output: Semantic Bottleneck Networks

arXiv: https://arxiv.org/abs/1907.10882 · HTML: https://mlsanity.com/paper/1907.10882/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2019.

## Built on by (most cited first, 30 of 49)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Promises and Pitfalls of Black-Box Concept Learning Models](https://mlsanity.com/paper/2106.13314/) (2021)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Concept Bottleneck Model with Additional Unsupervised Concepts](https://mlsanity.com/paper/2202.01459/) (2022)
- [GlanceNets: Interpretabile, Leak-proof Concept-based Models](https://mlsanity.com/paper/2205.15612/) (2022)
- [Editing a classifier by rewriting its prediction rules](https://mlsanity.com/paper/2112.01008/) (2021)
- [Concept-level Debugging of Part-Prototype Networks](https://mlsanity.com/paper/2205.15769/) (2022)
- [Learning Bottleneck Concepts in Image Classification](https://mlsanity.com/paper/2304.10131/) (2023)
- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [Interpreting Pretrained Language Models via Concept Bottlenecks](https://mlsanity.com/paper/2311.05014/) (2023)
- [Towards a Collective Agenda on AI for Earth Science Data Analysis](https://mlsanity.com/paper/2104.05107/) (2021)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [Concept Learning for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.12232/) (2023)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [Editable User Profiles for Controllable Text Recommendation](https://mlsanity.com/paper/2304.04250/) (2023)
- [Keep CALM and Improve Visual Feature Attribution](https://mlsanity.com/paper/2106.07861/) (2021)
- [CLIP-QDA: An Explainable Concept Bottleneck Model](https://mlsanity.com/paper/2312.00110/) (2023)
- [Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention](https://mlsanity.com/paper/2312.15033/) (2023)
- [Show and Tell: Visually Explainable Deep Neural Nets via Spatially-Aware Concept Bottleneck Models](https://mlsanity.com/paper/2502.20134/) (2025)
- [Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning](https://mlsanity.com/paper/2404.03323/) (2024)
- [Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces](https://mlsanity.com/paper/2311.14435/) (2023)
- [Large Language Models are Interpretable Learners](https://mlsanity.com/paper/2406.17224/) (2024)
- [Interpretable Prognostics with Concept Bottleneck Models](https://mlsanity.com/paper/2405.17575/) (2024)
- [Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models](https://mlsanity.com/paper/2406.12649/) (2024)
- [Enhancing Interpretable Image Classification Through LLM Agents and Conditional Concept Bottleneck Models](https://mlsanity.com/paper/2506.01334/) (2025)
- [Hierarchical Explanations for Video Action Recognition](https://mlsanity.com/paper/2301.00436/) (2023)
- [CAT: Interpretable Concept-based Taylor Additive Models](https://mlsanity.com/paper/2406.17931/) (2024)

## Abstract (arXiv)

Today's deep learning systems deliver high performance based on end-to-end training. While they deliver strong performance, these systems are hard to interpret. To address this issue, we propose Semantic Bottleneck Networks (SBN): deep networks with semantically interpretable intermediate layers that all downstream results are based on. As a consequence, the analysis on what the final prediction is based on is transparent to the engineer and failure cases and modes can be analyzed and avoided by high-level reasoning. We present a case study on street scene segmentation to demonstrate the feasibility and power of SBN. In particular, we start from a well performing classic deep network which we adapt to house a SB-Layer containing task related semantic concepts (such as object-parts and materials). Importantly, we can recover state of the art performance despite a drastic dimensionality reduction from 1000s (non-semantic feature) to 10s (semantic concept) channels. Additionally we show how the activations of the SB-Layer can be used for both the interpretation of failure cases of the network as well as for confidence prediction of the resulting output. For the first time, e.g., we show interpretable segmentation results for most predictions at over 99% accuracy.