# ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts

arXiv: https://arxiv.org/abs/2301.12276 · HTML: https://mlsanity.com/paper/2301.12276/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 81 references that are in the map.
- First posted on arXiv 28 Jan 2023.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)

## Built on by (most cited first, 30 of 33)

- [Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanations](https://mlsanity.com/paper/2308.08162/) (2023)
- [ICICLE: Interpretable Class Incremental Continual Learning](https://mlsanity.com/paper/2303.07811/) (2023)
- [Pixel-Grounded Prototypical Part Networks](https://mlsanity.com/paper/2309.14531/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision](https://mlsanity.com/paper/2405.14331/) (2024)
- [Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Survey](https://mlsanity.com/paper/2405.01636/) (2024)
- [This Looks Better than That: Better Interpretable Models with ProtoPNeXt](https://mlsanity.com/paper/2406.14675/) (2024)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [XEdgeAI: A Human-centered Industrial Inspection Framework with Data-centric Explainable Edge AI Approach](https://mlsanity.com/paper/2407.11771/) (2024)
- [Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models](https://mlsanity.com/paper/2407.16653/) (2024)
- [An Overview of Prototype Formulations for Interpretable Deep Learning](https://mlsanity.com/paper/2410.08925/) (2024)
- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)
- [Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation](https://mlsanity.com/paper/2409.09497/) (2024)
- [Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation](https://mlsanity.com/paper/2411.04607/) (2024)
- [SIDE: Sparse Information Disentanglement for Explainable Artificial Intelligence](https://mlsanity.com/paper/2507.19321/) (2025)
- [Interpretable Affordance Detection on 3D Point Clouds with Probabilistic Prototypes](https://mlsanity.com/paper/2504.18355/) (2025)
- [LPD: Learnable Prototypes with Diversity Regularization for Weakly Supervised Histopathology Segmentation](https://mlsanity.com/paper/2512.05922/) (2025)
- [ProtoP-OD: Explainable Object Detection with Prototypical Parts](https://mlsanity.com/paper/2402.19142/) (2024)
- [MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification](https://mlsanity.com/paper/2404.08917/) (2024)
- [COMIX: Compositional Explanations using Prototypes](https://mlsanity.com/paper/2501.06059/) (2025)
- [Comprehensive Evaluation of Prototype Neural Networks](https://mlsanity.com/paper/2507.06819/) (2025)
- [T-DuMpRa: Teacher-guided Dual-path Multi-prototype Retrieval Augmented framework for fine-grained medical image classification](https://mlsanity.com/paper/2604.17360/) (2026)
- [Leveraging Habitat Information for Fine-grained Bird Identification](https://mlsanity.com/paper/2312.14999/) (2023)
- [CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalities](https://mlsanity.com/paper/2506.08690/) (2025)
- [Deeply Dual Supervised learning for melanoma recognition](https://mlsanity.com/paper/2508.01994/) (2025)
- [Edge Detection for Organ Boundaries via Top Down Refinement and SubPixel Upsampling](https://mlsanity.com/paper/2508.06805/) (2025)
- [DualResolution Residual Architecture with Artifact Suppression for Melanocytic Lesion Segmentation](https://mlsanity.com/paper/2508.06816/) (2025)
- [VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagation](https://mlsanity.com/paper/2508.06819/) (2025)
- [ProtoQuant: Quantization of Prototypical Parts For General and Fine-Grained Image Classification](https://mlsanity.com/paper/2602.06592/) (2026)
- [RamanSeg: Interpretability-driven Deep Learning on Raman Spectra for Cancer Diagnosis](https://mlsanity.com/paper/2602.18119/) (2026)

## Abstract (arXiv)

We introduce ProtoSeg, a novel model for interpretable semantic image segmentation, which constructs its predictions using similar patches from the training set. To achieve accuracy comparable to baseline methods, we adapt the mechanism of prototypical parts and introduce a diversity loss function that increases the variety of prototypes within each class. We show that ProtoSeg discovers semantic concepts, in contrast to standard segmentation models. Experiments conducted on Pascal VOC and Cityscapes datasets confirm the precision and transparency of the presented method.