# TorchEsegeta: Framework for Interpretability and Explainability of Image-based Deep Learning Models

arXiv: https://arxiv.org/abs/2110.08429 · HTML: https://mlsanity.com/paper/2110.08429/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 30 of 34)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Principles and Practice of Explainable Machine Learning](https://mlsanity.com/paper/2009.11698/) (2020)
- [CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation](https://mlsanity.com/paper/2009.10549/) (2020)
- [How Important Is a Neuron?](https://mlsanity.com/paper/1805.12233/) (2018)
- [Interpretability and Explainability: A Machine Learning Zoo Mini-tour](https://mlsanity.com/paper/2012.01805/) (2020)
- [Evolving Normalization-Activation Layers](https://mlsanity.com/paper/2004.02967/) (2020)
- [Visual Analytics for Explainable Deep Learning](https://mlsanity.com/paper/1804.02527/) (2018)
- [Influence-Directed Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1802.03788/) (2018)
- [Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications](https://mlsanity.com/paper/2003.07631/) (2020)
- [Explainable Artificial Intelligence and Machine Learning: A reality rooted perspective](https://mlsanity.com/paper/2001.09464/) (2020)

## Built on by (most cited first, 4 of 4)

- [Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Survey](https://mlsanity.com/paper/2405.01636/) (2024)
- [Weakly-supervised segmentation using inherently-explainable classification models and their application to brain tumour classification](https://mlsanity.com/paper/2206.05148/) (2022)
- [New Insights into Automatic Treatment Planning for Cancer Radiotherapy Using Explainable Artificial Intelligence](https://mlsanity.com/paper/2508.14229/) (2025)
- [Estimating Total Lung Volume from Pixel-level Thickness Maps of Chest Radiographs Using Deep Learning](https://mlsanity.com/paper/2110.12509/) (2021)

## Abstract (arXiv)

Clinicians are often very sceptical about applying automatic image processing approaches, especially deep learning based methods, in practice. One main reason for this is the black-box nature of these approaches and the inherent problem of missing insights of the automatically derived decisions. In order to increase trust in these methods, this paper presents approaches that help to interpret and explain the results of deep learning algorithms by depicting the anatomical areas which influence the decision of the algorithm most. Moreover, this research presents a unified framework, TorchEsegeta, for applying various interpretability and explainability techniques for deep learning models and generate visual interpretations and explanations for clinicians to corroborate their clinical findings. In addition, this will aid in gaining confidence in such methods. The framework builds on existing interpretability and explainability techniques that are currently focusing on classification models, extending them to segmentation tasks. In addition, these methods have been adapted to 3D models for volumetric analysis. The proposed framework provides methods to quantitatively compare visual explanations using infidelity and sensitivity metrics. This framework can be used by data scientists to perform post-hoc interpretations and explanations of their models, develop more explainable tools and present the findings to clinicians to increase their faith in such models. The proposed framework was evaluated based on a use case scenario of vessel segmentation models trained on Time-of-fight (TOF) Magnetic Resonance Angiogram (MRA) images of the human brain. Quantitative and qualitative results of a comparative study of different models and interpretability methods are presented. Furthermore, this paper provides an extensive overview of several existing interpretability and explainability methods.