mlsanityData to 7 Oct 2026

Interpretable Image Classification with Adaptive Prototype-based Vision Transformers

arXiv 2410.20722 ↗First posted 28 Oct 2024cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

We present ProtoViT, a method for interpretable image classification combining deep learning and case-based reasoning. This method classifies an image by comparing it to a set of learned prototypes, providing explanations of the form ``this looks like that.'' In our model, a prototype consists of textitparts, which can deform over irregular geometries to create a better comparison between images. Unlike existing models that rely on Convolutional Neural Network (CNN) backbones and spatially rigid prototypes, our model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes that not only accommodate geometric variations of objects but also provide coherent and clear prototypical feature representations with an adaptive number of prototypical parts. Our experiments show that our model can generally achieve higher performance than the existing prototype based models. Our comprehensive analyses ensure that the prototypes are consistent and the interpretations are faithful.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  6. 2017Deformable Convolutional Networkscited by 1,481
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2019Attention is not Explanationcited by 916
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2021Going deeper with Image Transformerscited by 759
  12. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
25 more
  1. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 666
  2. 2019Attention is not not Explanationcited by 629
  3. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  4. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  5. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  6. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 213
  7. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  8. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  9. 2021TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localizationcited by 122
  10. 2021Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypescited by 120
  11. 2019The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanationscited by 97
  12. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  13. 2020ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discoverycited by 89
  14. 2021IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammographycited by 72
  15. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  16. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  17. 2023Learning Support and Trivial Prototypes for Interpretable Image Classificationcited by 38
  18. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 37
  19. 2023This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizationscited by 36
  20. 2016Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modelingcited by 28
  21. 2023Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanationscited by 27
  22. 2023Pixel-Grounded Prototypical Part Networkscited by 18
  23. 2023A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysiscited by 15
  24. 2022Knowledge Distillation to Ensemble Global and Interpretable Prototype-Based Mammogram Classification Modelscited by 11
  25. 2022Interpretable AI for relating brain structural and functional connectomescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 11.

25 papers.

Sort:

By area:

  1. 2026Interpretability Can Be Actionablecited by 5
  2. 2025This looks like what? Challenges and Future Research Directions for Part-Prototype Modelscited by 3
  3. 2025Birds look like cars: Adversarial analysis of intrinsically interpretable deep learningcited by 3
  4. 2025SIDE: Sparse Information Disentanglement for Explainable Artificial Intelligencecited by 3
  5. 2024A Robust Prototype-Based Network with Interpretable RBF Classifier Foundationscited by 2
  6. 2025Tell me why: Visual foundation models as self-explainable classifierscited by 2
  7. 2025From Pixels to Perception: Interpretable Predictions via Instance-wise Grouped Feature Selectioncited by 2
  8. 2025Debugging Concept Bottleneck Models through Removal and Retrainingcited by 2
  9. 2025Beyond Patches: Mining Interpretable Part-Prototypes for Explainable AIcited by 1
  10. 2025This EEG Looks Like These EEGs: Interpretable Interictal Epileptiform Discharge Detection With ProtoEEG-kNNcited by 1
  11. 2026Interpretable Vision Transformers in Image Classification via SVDAcited by 1
  12. 2026Prototype Transformer: Towards Language Model Architectures Interpretable by Designcited by 1
13 more
  1. 2026Hyperspherical Forward-Forward with Prototypical Representationscited by 1
  2. 2025DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalitiescited by 0
  3. 2025ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answeringcited by 0
  4. 2025ProtoMask: Segmentation-Guided Prototype Learningcited by 0
  5. 2025Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Predictioncited by 0
  6. 2025What You See is (Usually) What You Get: Multimodal Prototype Networks that Abstain from Expensive Modalitiescited by 0
  7. 2026ProtoQuant: Quantization of Prototypical Parts For General and Fine-Grained Image Classificationcited by 0
  8. 2026DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classificationcited by 0
  9. 2026ProtoTTA: Prototype-Guided Test-Time Adaptationcited by 0
  10. 2026Spatially Grounded Concept Bottleneck Models via Part-Factorized Attentioncited by 0
  11. 2026DINOv3-MIL: Per-Kidney Multi-Label Tumour and Cyst Detection from Foundation-Model Patch Tokens on KiTS23cited by 0
  12. 2026Prescriptive SVD-Inspired Attention via Spectral Energy Retentioncited by 0
  13. 2026Diverse by Design: Architectural Constraints for Prototype-Based Interpretabilitycited by 0