# Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations

arXiv: https://arxiv.org/abs/2609.38625 · HTML: https://mlsanity.com/paper/2609.38625/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Probabilistic Concept Bottleneck Models](https://mlsanity.com/paper/2306.01574/) (2023)
- [Interactive Concept Bottleneck Models](https://mlsanity.com/paper/2212.07430/) (2022)
- [Tight Certificates of Adversarial Robustness for Randomly Smoothed Classifiers](https://mlsanity.com/paper/1906.04948/) (2019)
- [Guided Variational Autoencoder for Disentanglement Learning](https://mlsanity.com/paper/2004.01255/) (2020)
- [Certified Robustness of Graph Neural Networks against Adversarial Structural Perturbation](https://mlsanity.com/paper/2008.10715/) (2020)
- [A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes](https://mlsanity.com/paper/2201.10766/) (2022)
- [Stochastic Concept Bottleneck Models](https://mlsanity.com/paper/2406.19272/) (2024)
- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [Enhancing Adversarial Training with Second-Order Statistics of Weights](https://mlsanity.com/paper/2203.06020/) (2022)
- [Adversarial Robustness of Deep Neural Networks: A Survey from a Formal Verification Perspective](https://mlsanity.com/paper/2206.12227/) (2022)
- [Language Guided Concept Bottleneck Models for Interpretable Continual Learning](https://mlsanity.com/paper/2503.23283/) (2025)
- [GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing](https://mlsanity.com/paper/2206.04310/) (2022)
- [Editable Concept Bottleneck Models](https://mlsanity.com/paper/2405.15476/) (2024)
- [Understanding and Enhancing Robustness of Concept-based Models](https://mlsanity.com/paper/2211.16080/) (2022)
- [Pixel-Grounded Prototypical Part Networks](https://mlsanity.com/paper/2309.14531/) (2023)
- [Evaluating Latent Space Robustness and Uncertainty of EEG-ML Models under Realistic Distribution Shifts](https://mlsanity.com/paper/2209.11233/) (2022)
- [Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders](https://mlsanity.com/paper/2505.16004/) (2025)
- [Enhancing Robust Fairness via Confusional Spectral Regularization](https://mlsanity.com/paper/2501.13273/) (2025)
- [Interpretable Generative Models through Post-hoc Concept Bottlenecks](https://mlsanity.com/paper/2503.19377/) (2025)
- [C-SENN: Contrastive Self-Explaining Neural Network](https://mlsanity.com/paper/2206.09575/) (2022)

## Abstract (arXiv)

Concept Bottleneck Models (CBMs) are designed to provide interpretable intermediate representations, yet how such bottlenecks affect robustness remains unclear, with existing studies reporting mixed and sometimes contradictory findings. We argue that these discrepancies arise from conflating different robustness notions and perturbation regimes, rather than from fundamental disagreements about CBMs themselves. To disentangle these factors, we introduce a generator-based evaluation framework that enables controlled comparisons between standard classifiers and CBMs under two distinct perturbation types: continuous geometric perturbations in latent space and discrete semantic interventions in concept space. Within this framework, we evaluate robustness both empirically, via prediction and concept-level sensitivity metrics, and certifiably, using randomized smoothing in latent and concept spaces. Across experiments, we reconcile previously conflicting findings by clarifying when, and in what sense, concept bottlenecks do or do not improve robustness. By further analyzing robustness under varying task conditions, including class semantic similarity and concept vocabulary size, we show that interpretability does not inherently confer robustness. Instead, concept bottlenecks shift where and how sensitivity manifests, revealing a nuanced interpretability robustness trade off that depends critically on the perturbation regime and task structure. Together, our results show that interpretability and robustness are distinct objectives: interpretable intermediate representations do not uniformly improve robustness, but instead redistribute sensitivity across perturbation spaces and model families.