# Deep neural network or dermatologist?

arXiv: https://arxiv.org/abs/1908.06612 · HTML: https://mlsanity.com/paper/1908.06612/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 20 of 20)

- [Explainable artificial intelligence (XAI) in deep learning-based medical image analysis](https://mlsanity.com/paper/2107.10912/) (2021)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [Towards Robust Classification Model by Counterfactual and Invariant Data Generation](https://mlsanity.com/paper/2106.01127/) (2021)
- [A survey, review, and future trends of skin lesion segmentation and classification](https://mlsanity.com/paper/2208.12232/) (2022)
- [ExAID: A Multimodal Explanation Framework for Computer-Aided Diagnosis of Skin Lesions](https://mlsanity.com/paper/2201.01249/) (2022)
- [Surrogate Model-Based Explainability Methods for Point Cloud NNs](https://mlsanity.com/paper/2107.13459/) (2021)
- [MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment](https://mlsanity.com/paper/2401.08527/) (2024)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging](https://mlsanity.com/paper/2109.09658/) (2021)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [Information Bottleneck Attribution for Visual Explanations of Diagnosis and Prognosis](https://mlsanity.com/paper/2104.02869/) (2021)
- [Deep Learning, Natural Language Processing, and Explainable Artificial Intelligence in the Biomedical Domain](https://mlsanity.com/paper/2202.12678/) (2022)
- [CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray Segmentation](https://mlsanity.com/paper/2102.10484/) (2021)
- [Deep Learning Based Decision Support for Medicine -- A Case Study on Skin Cancer Diagnosis](https://mlsanity.com/paper/2103.05112/) (2021)
- [Disentangled representations: towards interpretation of sex determination from hip bone](https://mlsanity.com/paper/2112.09414/) (2021)
- [Dermatological Diagnosis Explainability Benchmark for Convolutional Neural Networks](https://mlsanity.com/paper/2302.12084/) (2023)
- [A Sneak Attack on Segmentation of Medical Images Using Deep Neural Network Classifiers](https://mlsanity.com/paper/2201.02771/) (2022)
- [Toward Accessible Dermatology: Skin Lesion Classification Using Deep Learning Models on Mobile-Acquired Images](https://mlsanity.com/paper/2509.04800/) (2025)
- [An End-to-End Deep Learning Framework for Arsenicosis Diagnosis Using Mobile-Captured Skin Images](https://mlsanity.com/paper/2509.08780/) (2025)

## Abstract (arXiv)

Deep learning techniques have proven high accuracy for identifying melanoma in digitised dermoscopic images. A strength is that these methods are not constrained by features that are pre-defined by human semantics. A down-side is that it is difficult to understand the rationale of the model predictions and to identify potential failure modes. This is a major barrier to adoption of deep learning in clinical practice. In this paper we ask if two existing local interpretability methods, Grad-CAM and Kernel SHAP, can shed light on convolutional neural networks trained in the context of melanoma detection. Our contributions are (i) we first explore the domain space via a reproducible, end-to-end learning framework that creates a suite of 30 models, all trained on a publicly available data set (HAM10000), (ii) we next explore the reliability of GradCAM and Kernel SHAP in this context via some basic sanity check experiments (iii) finally, we investigate a random selection of models from our suite using GradCAM and Kernel SHAP. We show that despite high accuracy, the models will occasionally assign importance to features that are not relevant to the diagnostic task. We also show that models of similar accuracy will produce different explanations as measured by these methods. This work represents first steps in bridging the gap between model accuracy and interpretability in the domain of skin cancer classification.