# This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations

arXiv: https://arxiv.org/abs/2310.18589 · HTML: https://mlsanity.com/paper/2310.18589/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes](https://mlsanity.com/paper/2111.15000/) (2021)
- [HIVE: Evaluating the Human Interpretability of Visual Explanations](https://mlsanity.com/paper/2112.03184/) (2021)
- [The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations](https://mlsanity.com/paper/1907.09294/) (2019)
- [Interpretable Image Classification with Differentiable Prototypes Assignment](https://mlsanity.com/paper/2112.02902/) (2021)
- [ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery](https://mlsanity.com/paper/2011.14340/) (2020)
- [IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography](https://mlsanity.com/paper/2103.12308/) (2021)
- [This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition](https://mlsanity.com/paper/2011.02863/) (2020)
- [Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling](https://mlsanity.com/paper/1612.06573/) (2016)

## Built on by (most cited first, 30 of 36)

- [Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations](https://mlsanity.com/paper/2401.14142/) (2024)
- [Interpretable Image Classification with Adaptive Prototype-based Vision Transformers](https://mlsanity.com/paper/2410.20722/) (2024)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision](https://mlsanity.com/paper/2405.14331/) (2024)
- [Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in Real-Time](https://mlsanity.com/paper/2503.01087/) (2025)
- [Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis](https://mlsanity.com/paper/2501.09333/) (2025)
- [Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognition](https://mlsanity.com/paper/2312.00092/) (2023)
- [This Looks Better than That: Better Interpretable Models with ProtoPNeXt](https://mlsanity.com/paper/2406.14675/) (2024)
- [AudioProtoPNet: An interpretable deep learning model for bird sound classification](https://mlsanity.com/paper/2404.10420/) (2024)
- [ProtoS-ViT: Visual foundation models for sparse self-explainable classifications](https://mlsanity.com/paper/2406.10025/) (2024)
- [Post-hoc Part-prototype Networks](https://mlsanity.com/paper/2406.03421/) (2024)
- [Geodesic Prototype Matching via Diffusion Maps for Interpretable Fine-Grained Recognition](https://mlsanity.com/paper/2509.17050/) (2025)
- [An Overview of Prototype Formulations for Interpretable Deep Learning](https://mlsanity.com/paper/2410.08925/) (2024)
- [ProtoECGNet: Case-Based Interpretable Deep Learning for Multi-Label ECG Classification with Contrastive Learning](https://mlsanity.com/paper/2504.08713/) (2025)
- [Locality-aware Concept Bottleneck Model](https://mlsanity.com/paper/2508.14562/) (2025)
- [This Probably Looks Exactly Like That: An Invertible Prototypical Network](https://mlsanity.com/paper/2407.12200/) (2024)
- [Achieving Domain-Independent Certified Robustness via Knowledge Continuity](https://mlsanity.com/paper/2411.01644/) (2024)
- [This looks like what? Challenges and Future Research Directions for Part-Prototype Models](https://mlsanity.com/paper/2502.09340/) (2025)
- [Personalized Interpretability -- Interactive Alignment of Prototypical Parts Networks](https://mlsanity.com/paper/2506.05533/) (2025)
- [Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts](https://mlsanity.com/paper/2503.06084/) (2025)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [Prototype-Grounded Concept Models for Verifiable Concept Alignment](https://mlsanity.com/paper/2604.16076/) (2026)
- [FPN-IAIA-BL: A Multi-Scale Interpretable Deep Learning Model for Classification of Mass Margins in Digital Mammography](https://mlsanity.com/paper/2406.06386/) (2024)
- [Exploiting Interpretable Capabilities with Concept-Enhanced Diffusion and Prototype Networks](https://mlsanity.com/paper/2410.18705/) (2024)
- [COMIX: Compositional Explanations using Prototypes](https://mlsanity.com/paper/2501.06059/) (2025)
- [Comprehensive Evaluation of Prototype Neural Networks](https://mlsanity.com/paper/2507.06819/) (2025)
- [What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits](https://mlsanity.com/paper/2409.02335/) (2024)
- [Bi-ICE: An Inner Interpretable Framework for Image Classification via Bi-directional Interactions between Concept and Input Embeddings](https://mlsanity.com/paper/2411.18645/) (2024)
- [Enhancing Job Salary Prediction with Disentangled Composition Effect Modeling: A Neural Prototyping Approach](https://mlsanity.com/paper/2503.12978/) (2025)

## Abstract (arXiv)

We present ProtoConcepts, a method for interpretable image classification combining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-based image classification uses a ``this looks like that'' reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes to make a final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one comparisons, where a single training image patch serves as a prototype to compare with a part of our test image. With these single-image comparisons, it can often be difficult to identify the underlying concept being compared (e.g., ``is it comparing the color or the shape?''). Our proposed method modifies the architecture of prototype-based networks to instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., ``the test image and the related training patches are all the same shade of blue''), and allows our model to create richer, more interpretable visual explanations. Our experiments show that our ``this looks like those'' reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.