# SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis

arXiv: https://arxiv.org/abs/2302.00785 · HTML: https://mlsanity.com/paper/2302.00785/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [Disparities in Dermatology AI: Assessments Using Diverse Clinical Images](https://mlsanity.com/paper/2111.08006/) (2021)
- [MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts](https://mlsanity.com/paper/2202.06523/) (2022)
- [Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2106.12723/) (2021)
- [Deep Learning Predicts Hip Fracture using Confounding Patient and Healthcare Variables](https://mlsanity.com/paper/1811.03695/) (2018)
- [On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors](https://mlsanity.com/paper/2005.02000/) (2020)
- [Probing artificial neural networks: insights from neuroscience](https://mlsanity.com/paper/2104.08197/) (2021)
- [DermX: an end-to-end framework for explainable automated dermatological diagnosis](https://mlsanity.com/paper/2202.06956/) (2022)

## Built on by (most cited first, 30 of 44)

- [A Closer Look at the Intervention Procedure of Concept Bottleneck Models](https://mlsanity.com/paper/2302.14260/) (2023)
- [Auxiliary Losses for Learning Generalizable Concept-based Models](https://mlsanity.com/paper/2311.11108/) (2023)
- [Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decision](https://mlsanity.com/paper/2303.00885/) (2023)
- [WBCAtt: A White Blood Cell Dataset Annotated with Detailed Morphological Attributes](https://mlsanity.com/paper/2306.13531/) (2023)
- [XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization](https://mlsanity.com/paper/2403.09410/) (2024)
- [SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model](https://mlsanity.com/paper/2304.10691/) (2023)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [Integrating Clinical Knowledge into Concept Bottleneck Models](https://mlsanity.com/paper/2407.06600/) (2024)
- [MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks](https://mlsanity.com/paper/2505.06152/) (2025)
- [Path to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest Cost](https://mlsanity.com/paper/2306.10765/) (2023)
- [MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment](https://mlsanity.com/paper/2505.09372/) (2025)
- [Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis](https://mlsanity.com/paper/2404.05997/) (2024)
- [Describe Anything in Medical Images](https://mlsanity.com/paper/2505.05804/) (2025)
- [Intrinsic Self-Supervision for Data Quality Audits](https://mlsanity.com/paper/2305.17048/) (2023)
- [Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets](https://mlsanity.com/paper/2401.14497/) (2024)
- [Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosis](https://mlsanity.com/paper/2406.19130/) (2024)
- [Towards Reliable Dermatology Evaluation Benchmarks](https://mlsanity.com/paper/2309.06961/) (2023)
- [SkinCaRe: A Multimodal Dermatology Dataset Annotated with Medical Caption and Chain-of-Thought Reasoning](https://mlsanity.com/paper/2405.18004/) (2024)
- [Towards Multi-dimensional Explanation Alignment for Medical Classification](https://mlsanity.com/paper/2410.21494/) (2024)
- [Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales](https://mlsanity.com/paper/2411.00132/) (2024)
- [PASSION for Dermatology: Bridging the Diversity Gap with Pigmented Skin Images from Sub-Saharan Africa](https://mlsanity.com/paper/2411.04584/) (2024)
- [SegPrompt: Using Segmentation Map as a Better Prompt to Finetune Deep Models for Kidney Stone Classification](https://mlsanity.com/paper/2303.08303/) (2023)
- [DermSynth3D: Synthesis of in-the-wild Annotated Dermatology Images](https://mlsanity.com/paper/2305.12621/) (2023)
- [Visual Objectification in Films: Towards a New AI Task for Video Interpretation](https://mlsanity.com/paper/2401.13296/) (2024)
- [CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification](https://mlsanity.com/paper/2501.12266/) (2025)
- [Learning to Be A Doctor: Searching for Effective Medical Agent Architectures](https://mlsanity.com/paper/2504.11301/) (2025)
- [Concept-Based Unsupervised Domain Adaptation](https://mlsanity.com/paper/2505.05195/) (2025)
- [Learning Concept-Driven Logical Rules for Interpretable and Generalizable Medical Image Classification](https://mlsanity.com/paper/2505.14049/) (2025)
- [I Bet You Did Not Mean That: Testing Semantic Importance via Betting](https://mlsanity.com/paper/2405.19146/) (2024)
- [Concept Complement Bottleneck Model for Interpretable Medical Image Diagnosis](https://mlsanity.com/paper/2410.15446/) (2024)

## Abstract (arXiv)

For the deployment of artificial intelligence (AI) in high-risk settings, such as healthcare, methods that provide interpretability/explainability or allow fine-grained error analysis are critical. Many recent methods for interpretability/explainability and fine-grained error analysis use concepts, which are meta-labels that are semantically meaningful to humans. However, there are only a few datasets that include concept-level meta-labels and most of these meta-labels are relevant for natural images that do not require domain expertise. Densely annotated datasets in medicine focused on meta-labels that are relevant to a single disease such as melanoma. In dermatology, skin disease is described using an established clinical lexicon that allows clinicians to describe physical exam findings to one another. To provide a medical dataset densely annotated by domain experts with annotations useful across multiple disease processes, we developed SkinCon: a skin disease dataset densely annotated by dermatologists. SkinCon includes 3230 images from the Fitzpatrick 17k dataset densely annotated with 48 clinical concepts, 22 of which have at least 50 images representing the concept. The concepts used were chosen by two dermatologists considering the clinical descriptor terms used to describe skin lesions. Examples include "plaque", "scale", and "erosion". The same concepts were also used to label 656 skin disease images from the Diverse Dermatology Images dataset, providing an additional external dataset with diverse skin tone representations. We review the potential applications for the SkinCon dataset, such as probing models, concept-based explanations, and concept bottlenecks. Furthermore, we use SkinCon to demonstrate two of these use cases: debugging mistakes of an existing dermatology AI model with concepts and developing interpretable models with post-hoc concept bottleneck models.