# SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model arXiv: https://arxiv.org/abs/2304.10691 ยท HTML: https://mlsanity.com/paper/2304.10691/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 12 of its 70 references that are in the map. - First posted on arXiv 21 Apr 2023. ## Builds on (most cited first, 12 of 12) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022) - [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023) - [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019) - [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023) - [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023) - [Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT](https://mlsanity.com/paper/2304.04920/) (2023) - [Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI](https://mlsanity.com/paper/2302.11571/) (2023) ## Built on by (most cited first, 18 of 18) - [Large Language Models for Medicine: A Survey](https://mlsanity.com/paper/2405.13055/) (2024) - [GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis](https://mlsanity.com/paper/2411.16778/) (2024) - [MedVLThinker: Simple Baselines for Multimodal Medical Reasoning](https://mlsanity.com/paper/2508.02669/) (2025) - [Large language models in medicine: the potentials and pitfalls](https://mlsanity.com/paper/2309.00087/) (2023) - [MuseChat: A Conversational Music Recommendation System for Videos](https://mlsanity.com/paper/2310.06282/) (2023) - [MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queries](https://mlsanity.com/paper/2401.01596/) (2024) - [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025) - [MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output](https://mlsanity.com/paper/2510.10011/) (2025) - [Path to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest Cost](https://mlsanity.com/paper/2306.10765/) (2023) - [Automated Bioinformatics Analysis via AutoBA](https://mlsanity.com/paper/2309.03242/) (2023) - [Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation](https://mlsanity.com/paper/2512.16201/) (2025) - [HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding](https://mlsanity.com/paper/2506.05831/) (2025) - [Exploiting LMM-based knowledge for image classification tasks](https://mlsanity.com/paper/2406.03071/) (2024) - [Architecting Clinical Collaboration: Multi-Agent Reasoning Systems for Multimodal Medical VQA](https://mlsanity.com/paper/2507.05520/) (2025) - [Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks](https://mlsanity.com/paper/2511.03328/) (2025) - [A Medical Multimodal Large Language Model for Pediatric Pneumonia](https://mlsanity.com/paper/2409.02608/) (2024) - [LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration](https://mlsanity.com/paper/2603.24696/) (2026) - [CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis](https://mlsanity.com/paper/2605.08493/) (2026) ## Abstract (arXiv) Skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases, impacting a considerable portion of the population. Nonetheless, the field of dermatology diagnosis faces three significant hurdles. Firstly, there is a shortage of dermatologists accessible to diagnose patients, particularly in rural regions. Secondly, accurately interpreting skin disease images poses a considerable challenge. Lastly, generating patient-friendly diagnostic reports is usually a time-consuming and labor-intensive task for dermatologists. To tackle these challenges, we present SkinGPT-4, which is the world's first interactive dermatology diagnostic system powered by an advanced visual large language model. SkinGPT-4 leverages a fine-tuned version of MiniGPT-4, trained on an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes. We designed a two-step training process to allow SkinGPT to express medical features in skin disease images with natural language and make accurate diagnoses of the types of skin diseases. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identifies the characteristics and categories of the skin conditions, performs in-depth analysis, and provides interactive treatment recommendations. Meanwhile, SkinGPT-4's local deployment capability and commitment to user privacy also render it an appealing choice for patients in search of a dependable and precise diagnosis of their skin ailments. To demonstrate the robustness of SkinGPT-4, we conducted quantitative evaluations on 150 real-life cases, which were independently reviewed by certified dermatologists, and showed that SkinGPT-4 could provide accurate diagnoses of skin diseases.