# Contextual Semantic Interpretability

arXiv: https://arxiv.org/abs/2009.08720 · HTML: https://mlsanity.com/paper/2009.08720/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2020.

## Built on by (most cited first, 14 of 14)

- [Coarse-to-Fine Concept Bottleneck Models](https://mlsanity.com/paper/2310.02116/) (2023)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [Towards a Collective Agenda on AI for Earth Science Data Analysis](https://mlsanity.com/paper/2104.05107/) (2021)
- [Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representations](https://mlsanity.com/paper/2112.02290/) (2021)
- [Sparse Linear Concept Discovery Models](https://mlsanity.com/paper/2308.10782/) (2023)
- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning](https://mlsanity.com/paper/2404.03323/) (2024)
- [ELUDE: Generating interpretable explanations via a decomposition into labelled and unlabelled features](https://mlsanity.com/paper/2206.07690/) (2022)
- [DISCOVER: Making Vision Networks Interpretable via Competition and Dissection](https://mlsanity.com/paper/2310.04929/) (2023)
- [Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation](https://mlsanity.com/paper/2409.09497/) (2024)
- [UFO: A unified method for controlling Understandability and Faithfulness Objectives in concept-based explanations for CNNs](https://mlsanity.com/paper/2303.15632/) (2023)
- [Deep Learning and Earth Observation to Support the Sustainable Development Goals](https://mlsanity.com/paper/2112.11367/) (2021)
- [Clarity: The Flexibility-Interpretability Trade-Off in Sparsity-aware Concept Bottleneck Models](https://mlsanity.com/paper/2601.21944/) (2026)

## Abstract (arXiv)

Convolutional neural networks (CNN) are known to learn an image representation that captures concepts relevant to the task, but do so in an implicit way that hampers model interpretability. However, one could argue that such a representation is hidden in the neurons and can be made explicit by teaching the model to recognize semantically interpretable attributes that are present in the scene. We call such an intermediate layer a semantic bottleneck. Once the attributes are learned, they can be re-combined to reach the final decision and provide both an accurate prediction and an explicit reasoning behind the CNN decision. In this paper, we look into semantic bottlenecks that capture context: we want attributes to be in groups of a few meaningful elements and participate jointly to the final decision. We use a two-layer semantic bottleneck that gathers attributes into interpretable, sparse groups, allowing them contribute differently to the final output depending on the context. We test our contextual semantic interpretable bottleneck (CSIB) on the task of landscape scenicness estimation and train the semantic interpretable bottleneck using an auxiliary database (SUN Attributes). Our model yields in predictions as accurate as a non-interpretable baseline when applied to a real-world test set of Flickr images, all while providing clear and interpretable explanations for each prediction.