# Concept-level Debugging of Part-Prototype Networks

arXiv: https://arxiv.org/abs/2205.15769 · HTML: https://mlsanity.com/paper/2205.15769/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 66 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Interpretations are useful: penalizing explanations to align neural networks with prior knowledge](https://mlsanity.com/paper/1909.13584/) (2019)
- [Making deep neural networks right for the right scientific reasons by interacting with their explanations](https://mlsanity.com/paper/2001.05371/) (2020)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- ["How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations](https://mlsanity.com/paper/1911.06473/) (2019)
- [COVID-19 Image Data Collection](https://mlsanity.com/paper/2003.11597/) (2020)
- [Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations](https://mlsanity.com/paper/2011.12854/) (2020)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)
- [ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery](https://mlsanity.com/paper/2011.14340/) (2020)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)

## Built on by (most cited first, 30 of 52)

- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks](https://mlsanity.com/paper/2212.05946/) (2022)
- [Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsal](https://mlsanity.com/paper/2302.01242/) (2023)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Interpretable Image Classification with Adaptive Prototype-based Vision Transformers](https://mlsanity.com/paper/2410.20722/) (2024)
- [Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigation](https://mlsanity.com/paper/2412.05152/) (2024)
- [Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents](https://mlsanity.com/paper/2401.05821/) (2024)
- [Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning](https://mlsanity.com/paper/2309.07742/) (2023)
- [Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery](https://mlsanity.com/paper/2410.19213/) (2024)
- [From Hope to Safety: Unlearning Biases of Deep Models via Gradient Penalization in Latent Space](https://mlsanity.com/paper/2308.09437/) (2023)
- [A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts](https://mlsanity.com/paper/2406.10368/) (2024)
- [Pixel-Grounded Prototypical Part Networks](https://mlsanity.com/paper/2309.14531/) (2023)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [If Concept Bottlenecks are the Question, are Foundation Models the Answer?](https://mlsanity.com/paper/2504.19774/) (2025)
- [Understanding Inter-Concept Relationships in Concept-Based Models](https://mlsanity.com/paper/2405.18217/) (2024)
- [Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in Real-Time](https://mlsanity.com/paper/2503.01087/) (2025)
- [Leakage and Interpretability in Concept-Based Models](https://mlsanity.com/paper/2504.14094/) (2025)
- [On the Interpretability of Part-Prototype Based Classifiers: A Human Centric Analysis](https://mlsanity.com/paper/2310.06966/) (2023)
- [Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognition](https://mlsanity.com/paper/2312.00092/) (2023)
- [Interactive Medical Image Analysis with Concept-based Similarity Reasoning](https://mlsanity.com/paper/2503.06873/) (2025)
- [Conditional computation in neural networks: principles and research trends](https://mlsanity.com/paper/2403.07965/) (2024)
- [Where is the Truth? The Risk of Getting Confounded in a Continual World](https://mlsanity.com/paper/2402.06434/) (2024)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort](https://mlsanity.com/paper/2407.08947/) (2024)
- [SLIM: Spuriousness Mitigation with Minimal Human Annotations](https://mlsanity.com/paper/2407.05594/) (2024)
- [Improving Concept Alignment in Vision-Language Concept Bottleneck Models](https://mlsanity.com/paper/2405.01825/) (2024)

## Abstract (arXiv)

Part-prototype Networks (ProtoPNets) are concept-based classifiers designed to achieve the same performance as black-box models without compromising transparency. ProtoPNets compute predictions based on similarity to class-specific part-prototypes learned to recognize parts of training examples, making it easy to faithfully determine what examples are responsible for any target prediction and why. However, like other models, they are prone to picking up confounders and shortcuts from the data, thus suffering from compromised prediction accuracy and limited generalization. We propose ProtoPDebug, an effective concept-level debugger for ProtoPNets in which a human supervisor, guided by the model's explanations, supplies feedback in the form of what part-prototypes must be forgotten or kept, and the model is fine-tuned to align with this supervision. Our experimental evaluation shows that ProtoPDebug outperforms state-of-the-art debuggers for a fraction of the annotation cost. An online experiment with laypeople confirms the simplicity of the feedback requested to the users and the effectiveness of the collected feedback for learning confounder-free part-prototypes. ProtoPDebug is a promising tool for trustworthy interactive learning in critical applications, as suggested by a preliminary evaluation on a medical decision making task.