mlsanityData to 14 Oct 2026

Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning

arXiv 2503.08636 ↗First posted 11 Mar 2025cs.LG cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

A common belief is that intrinsically interpretable deep learning models ensure a correct, intuitive understanding of their behavior and offer greater robustness against accidental errors or intentional manipulation. However, these beliefs have not been comprehensively verified, and growing evidence casts doubt on them. In this paper, we highlight the risks related to overreliance and susceptibility to adversarial manipulation of these so-called "intrinsically (aka inherently) interpretable" models by design. We introduce two strategies for adversarial analysis with prototype manipulation and backdoor attacks against prototype-based networks, and discuss how concept bottleneck models defend against these attacks. Fooling the model's reasoning by exploiting its use of latent prototypes manifests the inherent uninterpretability of deep neural networks, leading to a false sense of security reinforced by a visual confirmation bias. The reported limitations of part-prototype networks put their trustworthiness and applicability into question, motivating further work on the robustness and alignment of (deep) interpretable models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 74 of its 106 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

74 of 106 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2022A ConvNet for the 2020scited by 3,579
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  8. 2020Concept Bottleneck Modelscited by 1,032
  9. 2021Going deeper with Image Transformerscited by 764
  10. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  11. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  12. 2017Interpretation of Neural Networks is Fragilecited by 444
38 more
  1. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  2. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  3. 2023Label-Free Concept Bottleneck Modelscited by 285
  4. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  5. 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
  6. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
  7. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 225
  8. 2019Explanations can be manipulated and geometry is to blamecited by 216
  9. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 214
  10. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  11. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
  12. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  13. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  14. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  15. 2020ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discoverycited by 89
  16. 2020Fairwashing Explanations with Off-Manifold Detergentcited by 72
  17. 2021This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networkscited by 65
  18. 2023A Closer Look at the Intervention Procedure of Concept Bottleneck Modelscited by 64
  19. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  20. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  21. 2020This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognitioncited by 50
  22. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 50
  23. 2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 48
  24. 2022Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capabilitycited by 46
  25. 2022Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networkscited by 37
  26. 2024Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretationscited by 33
  27. 2023B-cos Alignment for Inherently Interpretable CNNs and Vision Transformerscited by 31
  28. 2023Exploring and Interacting with the Set of Good Sparse Generalized Additive Modelscited by 30
  29. 2022SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretabilitycited by 28
  30. 2023Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanationscited by 27
  31. 2021The Role of Explainability in Assuring Safety of Machine Learning in Healthcarecited by 26
  32. 2024Interpretable Image Classification with Adaptive Prototype-based Vision Transformerscited by 25
  33. 2021ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classificationcited by 22
  34. 2022Understanding and Enhancing Robustness of Concept-based Modelscited by 22
  35. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
  36. 2024Position: Explain to Question not to Justifycited by 20
  37. 2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
  38. 2024LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Visioncited by 15

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Leakage and Interpretability in Concept-Based Modelscited by 13
  2. 2025Model Science: getting serious about verification, explanation and control of AI systemscited by 3
  3. 2026DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Revealing Vulnerabilities of Neural Networks in Parameter Learning and Defense Against Explanation-Aware Backdoors
  2. 2025Breaking the Illusion of Security via Interpretation: Interpretable Vision Transformer Systems under Attack
  3. 2026Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training
  4. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models
  5. 2025A Comprehensive Survey on Self-Interpretable Neural Networks
  6. 2026Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
  7. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  8. 2024Challenges in Mechanistically Interpreting Model Representations