mlsanityData to 14 Oct 2026

Towards Reliable Dermatology Evaluation Benchmarks

arXiv 2309.06961 ↗First posted 13 Sep 2023cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Benchmark datasets for digital dermatology unwittingly contain inaccuracies that reduce trust in model performance estimates. We propose a resource-efficient data-cleaning protocol to identify issues that escaped previous curation. The protocol leverages an existing algorithmic cleaning strategy and is followed by a confirmation process terminated by an intuitive stopping criterion. Based on confirmation by multiple dermatologists, we remove irrelevant samples and near duplicates and estimate the percentage of label errors in six dermatology image datasets for model evaluation promoted by the International Skin Imaging Collaboration. Along with this paper, we publish revised file lists for each dataset which should be used for model evaluation. Our work paves the way for more trustworthy performance assessment in digital dermatology.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 51 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

18 of 51 references.

References · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  6. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  7. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  8. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 185
  9. 2019Deep learning with noisy labels: exploring techniques and remedies in medical image analysiscited by 173
  10. 2017Deep Learning is Robust to Massive Label Noisecited by 165
  11. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  12. 2021Disparities in Dermatology AI: Assessments Using Diverse Clinical Imagescited by 134
6 more
  1. 2020PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphonescited by 118
  2. 2023SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysiscited by 44
  3. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  4. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  5. 2023Intrinsic Self-Supervision for Data Quality Auditscited by 8
  6. 2022Identifying the Context Shift between Test Benchmarks and Production Datacited by 5

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 1.

6 papers.

  1. 2024Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasetscited by 7
  2. 2024S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Imagescited by 6
  3. 2024PASSION for Dermatology: Bridging the Diversity Gap with Pigmented Skin Images from Sub-Saharan Africacited by 5
  4. 2024MpoxVLM: A Vision-Language Model for Diagnosing Skin Lesions from Mpox Virus Infectioncited by 3
  5. 2025CleanPatrick: A Benchmark for Image Data Cleaningcited by 2
  6. 2024Towards Scalable Foundation Models for Digital Dermatologycited by 1