mlsanityData to 14 Oct 2026

SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models

arXiv 2404.14755 ↗First posted 23 Apr 2024cs.MM cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

With the continuous advancement of vision language models (VLMs) technology, remarkable research achievements have emerged in the dermatology field, the fourth most prevalent human disease category. However, despite these advancements, VLM still faces explainable problems to user in diagnosis due to the inherent complexity of dermatological conditions, existing tools offer relatively limited support for user comprehension. We propose SkinGEN, a diagnosis-to-generation framework that leverages the stable diffusion(SD) model to generate reference demonstrations from diagnosis results provided by VLM, thereby enhancing the visual explainability for users. Through extensive experiments with Low-Rank Adaptation (LoRA), we identify optimal strategies for skin condition image generation. We conduct a user study with 32 participants evaluating both the system performance and explainability. Results demonstrate that SkinGEN significantly improves users' comprehension of VLM predictions and fosters increased trust in the diagnostic process. This work paves the way for more transparent and user-centric VLM applications in dermatology and beyond.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 57 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

29 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  3. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
14 more
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  2. 2023Larger language models do in-context learning differentlycited by 394
  3. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  4. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 185
  5. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178
  6. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  8. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  9. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 46
  10. 2023Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Imagescited by 45
  11. 2022Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesiscited by 42
  12. 2024Derm-T2IM: Harnessing Synthetic Skin Lesion Data via Stable Diffusion Models for Enhanced Skin Disease Classification using ViT and CNNcited by 15
  13. 2023PatFig: Generating Short and Long Captions for Patent Figurescited by 12
  14. 2023Visual Question Answering in the Medical Domaincited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narrativescited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
  2. 2025VL-MedGuide: A Visual-Linguistic Large Model for Intelligent and Explainable Skin Disease Auxiliary Diagnosis
  3. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks
  4. 2025LesionGen: A Concept-Guided Diffusion Model for Dermatology Image Synthesis
  5. 2026Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis
  6. 2024SkinCaRe: A Multimodal Dermatology Dataset Annotated with Medical Caption and Chain-of-Thought Reasoning
  7. 2025CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering
  8. 2026Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?