# Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets

arXiv: https://arxiv.org/abs/2401.14497 · HTML: https://mlsanity.com/paper/2401.14497/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 118 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 30 of 38)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2106.12723/) (2021)
- [Quantum Vision Transformers](https://mlsanity.com/paper/2209.08167/) (2022)
- [Federated Conformal Predictors for Distributed Uncertainty Quantification](https://mlsanity.com/paper/2305.17564/) (2023)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decision](https://mlsanity.com/paper/2303.00885/) (2023)
- [Improving dermatology classifiers across populations using images generated by large diffusion models](https://mlsanity.com/paper/2211.13352/) (2022)
- [Beyond Confidence: Reliable Models Should Also Consider Atypicality](https://mlsanity.com/paper/2305.18262/) (2023)
- [Detecting Melanoma Fairly: Skin Tone Detection and Debiasing for Skin Lesion Classification](https://mlsanity.com/paper/2202.02832/) (2022)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learning](https://mlsanity.com/paper/2208.10013/) (2022)
- [Label-Assemble: Leveraging Multiple Datasets with Partial Labels](https://mlsanity.com/paper/2109.12265/) (2021)
- [Skin Lesion Recognition with Class-Hierarchy Regularized Hyperbolic Embeddings](https://mlsanity.com/paper/2209.05842/) (2022)
- [Differentially Private Image Classification by Learning Priors from Random Processes](https://mlsanity.com/paper/2306.06076/) (2023)
- [Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images](https://mlsanity.com/paper/2202.12267/) (2022)
- [CIRCLe: Color Invariant Representation Learning for Unbiased Classification of Skin Lesions](https://mlsanity.com/paper/2208.13528/) (2022)
- [Dataset Growth in Medical Image Analysis Research](https://mlsanity.com/paper/1908.07765/) (2019)
- [Re-thinking Federated Active Learning based on Inter-class Diversity](https://mlsanity.com/paper/2303.12317/) (2023)
- [Achieve Fairness without Demographics for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2401.08066/) (2024)
- [MedRDF: A Robust and Retrain-Less Diagnostic Framework for Medical Pretrained Models Against Adversarial Attack](https://mlsanity.com/paper/2111.14564/) (2021)

## Built on by (most cited first, 7 of 7)

- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)
- [Identity-Free Deferral For Unseen Experts](https://mlsanity.com/paper/2502.10533/) (2025)
- [CNN and ViT Efficiency Study on Tiny ImageNet and DermaMNIST Datasets](https://mlsanity.com/paper/2505.08259/) (2025)
- [CleanPatrick: A Benchmark for Image Data Cleaning](https://mlsanity.com/paper/2505.11034/) (2025)
- [Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification](https://mlsanity.com/paper/2512.15249/) (2025)
- [Be Fair! Can Machine Learning Engineering Agents Adhere to Fairness Constraints?](https://mlsanity.com/paper/2606.04971/) (2026)
- [Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset](https://mlsanity.com/paper/2507.12961/) (2025)

## Abstract (arXiv)

The remarkable progress of deep learning in dermatological tasks has brought us closer to achieving diagnostic accuracies comparable to those of human experts. However, while large datasets play a crucial role in the development of reliable deep neural network models, the quality of data therein and their correct usage are of paramount importance. Several factors can impact data quality, such as the presence of duplicates, data leakage across train-test partitions, mislabeled images, and the absence of a well-defined test partition. In this paper, we conduct meticulous analyses of three popular dermatological image datasets: DermaMNIST, its source HAM10000, and Fitzpatrick17k, uncovering these data quality issues, measure the effects of these problems on the benchmark results, and propose corrections to the datasets. Besides ensuring the reproducibility of our analysis, by making our analysis pipeline and the accompanying code publicly available, we aim to encourage similar explorations and to facilitate the identification and addressing of potential data quality issues in other large datasets.