mlsanityData to 14 Oct 2026

Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets

arXiv 2401.14497 ↗First posted 25 Jan 2024cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The remarkable progress of deep learning in dermatological tasks has brought us closer to achieving diagnostic accuracies comparable to those of human experts. However, while large datasets play a crucial role in the development of reliable deep neural network models, the quality of data therein and their correct usage are of paramount importance. Several factors can impact data quality, such as the presence of duplicates, data leakage across train-test partitions, mislabeled images, and the absence of a well-defined test partition. In this paper, we conduct meticulous analyses of three popular dermatological image datasets: DermaMNIST, its source HAM10000, and Fitzpatrick17k, uncovering these data quality issues, measure the effects of these problems on the benchmark results, and propose corrections to the datasets. Besides ensuring the reproducibility of our analysis, by making our analysis pipeline and the accompanying code publicly available, we aim to encourage similar explorations and to facilitate the identification and addressing of potential data quality issues in other large datasets.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 38 of its 118 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

38 of 118 references.

References · 38

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  3. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 669
  4. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  5. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 225
  6. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  7. 2020MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysiscited by 206
  8. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  9. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 185
  10. 2019Deep Semantic Segmentation of Natural and Medical Images: A Reviewcited by 125
  11. 2019A deep learning system for differential diagnosis of skin diseasescited by 118
  12. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
26 more
  1. 2022Quantum Vision Transformerscited by 48
  2. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  3. 2023SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysiscited by 44
  4. 2023Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decisioncited by 27
  5. 2022Improving dermatology classifiers across populations using images generated by large diffusion modelscited by 25
  6. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25
  7. 2022Detecting Melanoma Fairly: Skin Tone Detection and Debiasing for Skin Lesion Classificationcited by 23
  8. 2022A Survey on Deep Learning for Skin Lesion Segmentationcited by 22
  9. 2022FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learningcited by 22
  10. 2021Label-Assemble: Leveraging Multiple Datasets with Partial Labelscited by 20
  11. 2022Skin Lesion Recognition with Class-Hierarchy Regularized Hyperbolic Embeddingscited by 19
  12. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  13. 2022Inflation of test accuracy due to data leakage in deep learning-based classification of OCT imagescited by 17
  14. 2022CIRCLe: Color Invariant Representation Learning for Unbiased Classification of Skin Lesionscited by 17
  15. 2019Dataset Growth in Medical Image Analysis Researchcited by 16
  16. 2023Re-thinking Federated Active Learning based on Inter-class Diversitycited by 14
  17. 2024Achieve Fairness without Demographics for Dermatological Disease Diagnosiscited by 10
  18. 2021MedRDF: A Robust and Retrain-Less Diagnostic Framework for Medical Pretrained Models Against Adversarial Attackcited by 9
  19. 2023Achieving Reliable and Fair Skin Lesion Diagnosis via Unsupervised Domain Adaptationcited by 9
  20. 2023Accelerating Batch Active Learning Using Continual Learning Techniquescited by 8
  21. 2023How You Split Matters: Data Leakage and Subject Characteristics Studies in Longitudinal Brain MRI Analysiscited by 8
  22. 2022Adversarial Auto-Augment with Label Preservation: A Representation Learning Principle Guided Approachcited by 6
  23. 2023Towards Reliable Dermatology Evaluation Benchmarkscited by 6
  24. 2023Are labels informative in semi-supervised learning? -- Estimating and leveraging the missing-data mechanismcited by 5
  25. 2022Multi-Sample zeta-mixup: Richer, More Realistic Synthetic Samples from a p-Series Interpolantcited by 1
  26. 2022A Knowledge-based Learning Framework for Self-supervised Pre-training Towards Enhanced Recognition of Biomedical Microscopy Imagescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024Mask of truth: model sensitivity to unexpected regions of medical imagescited by 6
  2. 2025Identity-Free Deferral For Unseen Expertscited by 4
  3. 2025CNN and ViT Efficiency Study on Tiny ImageNet and DermaMNIST Datasetscited by 4
  4. 2025CleanPatrick: A Benchmark for Image Data Cleaningcited by 2
  5. 2025Intersectional Fairness in Vision-Language Models for Medical Image Disease Classificationcited by 1
  6. 2026Be Fair! Can Machine Learning Engineering Agents Adhere to Fairness Constraints?cited by 1
  7. 2025Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Datasetcited by 0