mlsanityData to 14 Oct 2026

ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition

arXiv 2208.10431 ↗First posted 22 Aug 2022cs.CV cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

Prototypical part network (ProtoPNet) has drawn wide attention and boosted many follow-up studies due to its self-explanatory property for explainable artificial intelligence (XAI). However, when directly applying ProtoPNet on vision transformer (ViT) backbones, learned prototypes have a "distraction" problem: they have a relatively high probability of being activated by the background and pay less attention to the foreground. The powerful capability of modeling long-term dependency makes the transformer-based ProtoPNet hard to focus on prototypical parts, thus severely impairing its inherent interpretability. This paper proposes prototypical part transformer (ProtoPFormer) for appropriately and effectively applying the prototype-based method with ViTs for interpretable image recognition. The proposed method introduces global and local prototypes for capturing and highlighting the representative holistic and partial features of targets according to the architectural characteristics of ViTs. The global prototypes are adopted to provide the global view of objects to guide local prototypes to concentrate on the foreground while eliminating the influence of the background. Afterwards, local prototypes are explicitly supervised to concentrate on their respective prototypical visual parts, increasing the overall interpretability. Extensive experiments demonstrate that our proposed global and local prototypes can mutually correct each other and jointly make final decisions, which faithfully and transparently reason the decision-making processes associatively from the whole and local perspectives, respectively. Moreover, ProtoPFormer consistently achieves superior performance and visualization results over the state-of-the-art (SOTA) prototype-based baselines. Our code has been released at https://github.com/zju-vipa/ProtoPFormer.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 44 of its 63 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

44 of 63 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
32 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  2. 2017Random Erasing Data Augmentationcited by 1,239
  3. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  4. 2016Deep Networks with Stochastic Depthcited by 979
  5. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  6. 2020Quantifying Attention Flow in Transformerscited by 775
  7. 2021Going deeper with Image Transformerscited by 764
  8. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  10. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  11. 2019Is Attention Interpretable?cited by 389
  12. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  13. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  14. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 214
  15. 2017Interpretable Convolutional Neural Networkscited by 162
  16. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  17. 2021TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localizationcited by 123
  18. 2021Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypescited by 120
  19. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  20. 2019Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Surveycited by 93
  21. 2018Interpreting CNNs via Decision Treescited by 89
  22. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  23. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  24. 2017Deep Lattice Networks and Partial Monotonic Functionscited by 63
  25. 2017SPINE: SParse Interpretable Neural Embeddingscited by 55
  26. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  27. 2021This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagationcited by 43
  28. 2021LCTR: On Awakening the Local Continuity of Transformer for Weakly Supervised Object Localizationcited by 24
  29. 2022Proto2Proto: Can you recognize the car, the way I do?cited by 23
  30. 2019Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretationcited by 22
  31. 2022ViTOL: Vision Transformer for Weakly Supervised Object Localizationcited by 13
  32. 2018Visualizing Convolutional Neural Networks to Improve Decision Support for Skin Lesion Classificationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 18; 2025: 21; 2026: 10.

63 papers.

Sort:

By area:

  1. 2024Interpretable Image Classification with Adaptive Prototype-based Vision Transformerscited by 25
  2. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Modelscited by 24
  3. 2024On the Concept Trustworthiness in Concept Bottleneck Modelscited by 22
  4. 2024Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discoverycited by 21
  5. 2023Pixel-Grounded Prototypical Part Networkscited by 18
  6. 2024PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generationcited by 18
  7. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  8. 2024DCBM: Data-Efficient Visual Concept Bottleneck Modelscited by 16
  9. 2023The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifierscited by 15
  10. 2023A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysiscited by 15
  11. 2025Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanationcited by 15
  12. 2025Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysiscited by 12
38 more
  1. 2023Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognitioncited by 11
  2. 2025Interpretable Image Classification via Non-parametric Part Prototype Learningcited by 11
  3. 2023Concept Distillation: Leveraging Human-Centered Explanations for Model Improvementcited by 10
  4. 2024AudioProtoPNet: An interpretable deep learning model for bird sound classificationcited by 10
  5. 2023GEMTrans: A General, Echocardiography-based, Multi-Level Transformer Framework for Cardiovascular Diagnosiscited by 9
  6. 2024A survey on Concept-based Approaches For Model Improvementcited by 8
  7. 2024ProtoS-ViT: Visual foundation models for sparse self-explainable classificationscited by 8
  8. 2023Concept-Centric Transformers: Enhancing Model Interpretability through Object-Centric Concept Learning within a Shared Global Workspacecited by 7
  9. 2024PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneckcited by 7
  10. 2025FOCUS: Fused Observation of Channels for Unveiling Spectracited by 7
  11. 2022SASFormer: Transformers for Sparsely Annotated Semantic Segmentationcited by 6
  12. 2023MoViT: Memorizing Vision Transformers for Medical Image Analysiscited by 6
  13. 2024On the Evaluation Consistency of Attribution-based Explanationscited by 5
  14. 2024An Overview of Prototype Formulations for Interpretable Deep Learningcited by 5
  15. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4
  16. 2025SIDE: Sparse Information Disentanglement for Explainable Artificial Intelligencecited by 4
  17. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
  18. 2023Improving Prototypical Visual Explanations with Reward Reweighing, Reselection, and Retrainingcited by 3
  19. 2024Semantic Prototypes: Enhancing Transparency Without Black Boxescited by 3
  20. 2024Quantifying and Enabling the Interpretability of CLIP-like Modelscited by 3
  21. 2025This looks like what? Challenges and Future Research Directions for Part-Prototype Modelscited by 3
  22. 2025Birds look like cars: Adversarial analysis of intrinsically interpretable deep learningcited by 3
  23. 2023Masked Collaborative Contrast for Weakly Supervised Semantic Segmentationcited by 2
  24. 2024ComFe: An Interpretable Head for Vision Transformerscited by 2
  25. 2025Tell me why: Visual foundation models as self-explainable classifierscited by 2
  26. 2025Recent Advances in Medical Image Classificationcited by 2
  27. 2023Schema Inference for Interpretable Image Classificationcited by 1
  28. 2024ViTree: Single-path Neural Tree for Step-wise Interpretable Fine-grained Visual Categorizationcited by 1
  29. 2025COMIX: Compositional Explanations using Prototypescited by 1
  30. 2025Disentangling Visual Transformers: Patch-level Interpretability for Image Classificationcited by 1
  31. 2025Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Biascited by 1
  32. 2025Proto-FG3D: Prototype-based Interpretable Fine-Grained 3D Shape Classificationcited by 1
  33. 2025Comprehensive Evaluation of Prototype Neural Networkscited by 1
  34. 2025Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoderscited by 1
  35. 2026Do Neural Networks Preserve Case Structure? Case-Based Decomposition, Interpretation, and Decision Consistencycited by 1
  36. 2023Leveraging Habitat Information for Fine-grained Bird Identificationcited by 0
  37. 2023DXAI: Explaining Classification by Image Decompositioncited by 0
  38. 2024Bi-ICE: An Inner Interpretable Framework for Image Classification via Bi-directional Interactions between Concept and Input Embeddingscited by 0

50 most cited of 63 All 63 (JSON)