# Making Corgis Important for Honeycomb Classification: Adversarial Attacks on Concept-based Explainability Tools

arXiv: https://arxiv.org/abs/2110.07120 · HTML: https://mlsanity.com/paper/2110.07120/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Fooling Neural Network Interpretations via Adversarial Model Manipulation](https://mlsanity.com/paper/1902.02041/) (2019)
- [Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks](https://mlsanity.com/paper/1602.03616/) (2016)
- [Fairwashing Explanations with Off-Manifold Detergent](https://mlsanity.com/paper/2007.09969/) (2020)
- [Robust and Stable Black Box Explanations](https://mlsanity.com/paper/2011.06169/) (2020)
- [Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values](https://mlsanity.com/paper/1810.03307/) (2018)
- [On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors](https://mlsanity.com/paper/2005.02000/) (2020)
- [Regression Concept Vectors for Bidirectional Explanations in Histopathology](https://mlsanity.com/paper/1904.04520/) (2019)
- [Understanding Intra-Class Knowledge Inside CNN](https://mlsanity.com/paper/1507.02379/) (2015)
- [Concept-based model explanations for Electronic Health Records](https://mlsanity.com/paper/2012.02308/) (2020)
- [How to Manipulate CNNs to Make Them Lie: the GradCAM Case](https://mlsanity.com/paper/1907.10901/) (2019)

## Built on by (most cited first, 9 of 9)

- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [The Effect of Data Poisoning on Counterfactual Explanations](https://mlsanity.com/paper/2402.08290/) (2024)
- [GCAV: A Global Concept Activation Vector Framework for Cross-Layer Consistency in Interpretability](https://mlsanity.com/paper/2508.21197/) (2025)
- [ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AI](https://mlsanity.com/paper/2607.09649/) (2026)
- [A Unified Concept-Based System for Local, Global, and Misclassification Explanations](https://mlsanity.com/paper/2306.03531/) (2023)
- [The Anatomy of Adversarial Attacks: Concept-based XAI Dissection](https://mlsanity.com/paper/2403.16782/) (2024)
- [Attack logics, not outputs: Towards efficient robustification of deep neural networks by falsifying concept-based properties](https://mlsanity.com/paper/2510.03320/) (2025)
- [ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on](https://mlsanity.com/paper/2608.26083/) (2026)

## Abstract (arXiv)

Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. Concept-based interpretability techniques, which use a small set of human-interpretable concept exemplars in order to measure the influence of a concept on a model's internal representation of input, are an important thread in this line of research. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based interpretability methods: TCAV and faceted feature visualization. We show that by carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method. The attacks that we propose can either induce positive interpretations (polka dots are an important concept for a model when classifying zebras) or negative interpretations (stripes are not an important factor in identifying images of a zebra). Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.