# A Novel Multi-Task Model Imitating Dermatologists for Accurate Differential Diagnosis of Skin Diseases in Clinical Images

arXiv: https://arxiv.org/abs/2307.08308 · HTML: https://mlsanity.com/paper/2307.08308/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Feature Fusion Vision Transformer for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2107.02341/) (2021)
- [The impact of patient clinical information on automated skin cancer detection](https://mlsanity.com/paper/1909.12912/) (2019)
- [DermX: an end-to-end framework for explainable automated dermatological diagnosis](https://mlsanity.com/paper/2202.06956/) (2022)
- [Learning Differential Diagnosis of Skin Conditions with Co-occurrence Supervision using Graph Convolutional Networks](https://mlsanity.com/paper/2007.06666/) (2020)
- [PCCT: Progressive Class-Center Triplet Loss for Imbalanced Medical Image Classification](https://mlsanity.com/paper/2207.04793/) (2022)

## Built on by (most cited first, 6 of 6)

- [Integrating Clinical Knowledge into Concept Bottleneck Models](https://mlsanity.com/paper/2407.06600/) (2024)
- [Automated Genomic Interpretation via Concept Bottleneck Models for Medical Robotics](https://mlsanity.com/paper/2510.01618/) (2025)
- [Location embedding based pairwise distance learning for fine-grained diagnosis of urinary stones](https://mlsanity.com/paper/2407.00431/) (2024)
- [Multi-task Learning Approach for Intracranial Hemorrhage Prognosis](https://mlsanity.com/paper/2408.08784/) (2024)
- [Anisotropic Diffusion Probabilistic Model for Imbalanced Image Classification](https://mlsanity.com/paper/2409.14313/) (2024)
- [Towards Automated Differential Diagnosis of Skin Diseases Using Deep Learning and Imbalance-Aware Strategies](https://mlsanity.com/paper/2601.00286/) (2026)

## Abstract (arXiv)

Skin diseases are among the most prevalent health issues, and accurate computer-aided diagnosis methods are of importance for both dermatologists and patients. However, most of the existing methods overlook the essential domain knowledge required for skin disease diagnosis. A novel multi-task model, namely DermImitFormer, is proposed to fill this gap by imitating dermatologists' diagnostic procedures and strategies. Through multi-task learning, the model simultaneously predicts body parts and lesion attributes in addition to the disease itself, enhancing diagnosis accuracy and improving diagnosis interpretability. The designed lesion selection module mimics dermatologists' zoom-in action, effectively highlighting the local lesion features from noisy backgrounds. Additionally, the presented cross-interaction module explicitly models the complicated diagnostic reasoning between body parts, lesion attributes, and diseases. To provide a more robust evaluation of the proposed method, a large-scale clinical image dataset of skin diseases with significantly more cases than existing datasets has been established. Extensive experiments on three different datasets consistently demonstrate the state-of-the-art recognition performance of the proposed approach.