mlsanityData to 14 Oct 2026

PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis

arXiv 2506.10669 ↗First posted 12 Jun 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Background and Objective: Prototype-based methods improve interpretability by learning fine-grained part-prototypes; however, their visualization in the input pixel space is not always consistent with human-understandable biomarkers. In addition, well-known prototype-based approaches typically learn extremely granular prototypes that are less interpretable in medical imaging, where both the presence and extent of biomarkers and lesions are critical. Methods: To address these challenges, we propose PiPViT (Patch-based Visual Interpretable Prototypes), an inherently interpretable prototypical model for image recognition. Leveraging a vision transformer (ViT), PiPViT captures long-range dependencies among patches to learn robust, human-interpretable prototypes that approximate lesion extent only using image-level labels. Additionally, PiPViT benefits from contrastive learning and multi-resolution input processing, which enables effective localization of biomarkers across scales. Results: We evaluated PiPViT on retinal OCT image classification across four datasets, where it achieved competitive quantitative performance compared to state-of-the-art methods while delivering more meaningful explanations. Moreover, quantitative evaluation on a hold-out test set confirms that the learned prototypes are semantically and clinically relevant. We believe PiPViT can transparently explain its decisions and assist clinicians in understanding diagnostic outcomes. Github page: https://github.com/marziehoghbaie/PiPViT

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 78 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

35 of 78 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2020Quantifying Attention Flow in Transformerscited by 775
  8. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  9. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  10. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  11. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 365
  12. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
23 more
  1. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 214
  2. 2022FlexiViT: One Model for All Patch Sizescited by 152
  3. 2018Spreading vectors for similarity searchcited by 118
  4. 2019Towards Explainable Artificial Intelligencecited by 106
  5. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  6. 2021TransMed: Transformers Advance Multi-modal Medical Image Classificationcited by 86
  7. 2021Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methodscited by 83
  8. 2021IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammographycited by 72
  9. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  10. 2023FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcarecited by 60
  11. 2018Deep Learning Predicts Hip Fracture using Confounding Patient and Healthcare Variablescited by 50
  12. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  13. 2019Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCTcited by 42
  14. 2018OCTID: Optical Coherence Tomography Image Databasecited by 36
  15. 2023B-cos Alignment for Inherently Interpretable CNNs and Vision Transformerscited by 31
  16. 2023OCTDL: Optical Coherence Tomography Dataset for Image-Based Deep Learning Methodscited by 28
  17. 2021GasHis-Transformer: A Multi-scale Visual Transformer Approach for Gastric Histopathological Image Detectioncited by 26
  18. 2023Towards Evaluating Explanations of Vision Transformers for Medical Imagingcited by 23
  19. 2021ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classificationcited by 22
  20. 2023Interpreting and Correcting Medical Image Classification with PIP-Netcited by 11
  21. 2024This actually looks like that: Proto-BagNets for local and global interpretability-by-designcited by 7
  22. 2024Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networkscited by 3
  23. 2021KFWC: A Knowledge-Driven Deep Learning Model for Fine-grained Classification of Wet-AMDcited by 1

Built on by

Not cited yet.