mlsanityData to 14 Oct 2026

CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering

arXiv 2508.18430 ↗First posted 25 Aug 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have shown significant potential for medical tasks; however, their general-purpose nature can limit specialized diagnostic accuracy, and their large size poses substantial inference costs for real-world clinical deployment. To address these challenges, we introduce CLARIFY, a Specialist-Generalist framework for dermatological visual question answering (VQA). CLARIFY combines two components: (i) a lightweight, domain-trained image classifier (the Specialist) that provides fast and highly accurate diagnostic predictions, and (ii) a powerful yet compressed conversational VLM (the Generalist) that generates natural language explanations to user queries. In our framework, the Specialist's predictions directly guide the Generalist's reasoning, focusing it on the correct diagnostic path. This synergy is further enhanced by a knowledge graph-based retrieval module, which grounds the Generalist's responses in factual dermatological knowledge, ensuring both accuracy and reliability. This hierarchical design not only reduces diagnostic errors but also significantly improves computational efficiency. Experiments on our curated multimodal dermatology dataset demonstrate that CLARIFY achieves an 18% improvement in diagnostic accuracy over the strongest baseline, a fine-tuned, uncompressed single-line VLM, while reducing the average VRAM requirement and latency by at least 20% and 5%, respectively. These results indicate that a Specialist-Generalist system provides a practical and powerful paradigm for building lightweight, trustworthy, and clinically viable AI systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 38 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

29 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2024A Survey on LLM-as-a-Judgecited by 1,423
16 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  7. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  8. 2023Towards Generalist Biomedical AIcited by 289
  9. 2023Multimodal Large Language Models: A Surveycited by 195
  10. 2025MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilotcited by 83
  11. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  12. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 56
  13. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  14. 2025KGGen: Extracting Knowledge Graphs from Plain Text with Language Modelscited by 49
  15. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  16. 2024A Comprehensive Study on Quantization Techniques for Large Language Modelscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning
  2. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks
  3. 2026How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
  4. 2025GRPO++: Enhancing Dermatological Reasoning under Low Resource Settings
  5. 2025MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment
  6. 2026Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?
  7. 2026A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology
  8. 2025DermaVQA-DAS: Dermatology Assessment Schema (DAS) & Datasets for Closed-Ended Question Answering & Segmentation in Patient-Generated Dermatology Images