# Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis

arXiv: https://arxiv.org/abs/2608.03225 · HTML: https://mlsanity.com/paper/2608.03225/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Disparities in Dermatology AI: Assessments Using Diverse Clinical Images](https://mlsanity.com/paper/2111.08006/) (2021)
- [A Closer Look at the Intervention Procedure of Concept Bottleneck Models](https://mlsanity.com/paper/2302.14260/) (2023)
- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [A Multimodal Vision Foundation Model for Clinical Dermatology](https://mlsanity.com/paper/2410.15038/) (2024)
- [MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment](https://mlsanity.com/paper/2401.08527/) (2024)
- [Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images](https://mlsanity.com/paper/2508.00549/) (2025)
- [MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment](https://mlsanity.com/paper/2505.09372/) (2025)
- [Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularization](https://mlsanity.com/paper/2410.07901/) (2024)
- [Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosis](https://mlsanity.com/paper/2406.19130/) (2024)
- [A Theory-Inspired Framework for Few-Shot Cross-Modal Sketch Person Re-Identification](https://mlsanity.com/paper/2511.18677/) (2025)
- [MetaModulation: Learning Variational Feature Hierarchies for Few-Shot Learning with Fewer Tasks](https://mlsanity.com/paper/2305.10309/) (2023)
- [QUOTA: Quantifying Objects with Text-to-Image Models for Any Domain](https://mlsanity.com/paper/2411.19534/) (2024)
- [Integrating Clinical Knowledge Graphs and Gradient-Based Neural Systems for Enhanced Melanoma Diagnosis via the 7-Point Checklist](https://mlsanity.com/paper/2407.16822/) (2024)
- [Adaptive Test-Time Intervention for Concept Bottleneck Models](https://mlsanity.com/paper/2503.06730/) (2025)

## Abstract (arXiv)

Human-interpretable computer-aided diagnosis is crucial for clinical decision making. Concept-based models excel by providing transparent reasoning and enabling post-hoc, clinician-in-the-loop interventions. However, their rigid dataset-specific adaptation inherently restricts cross-site generalization. Applying them across diverse modalities, such as dermoscopic and clinical photographs, is challenging due to heterogeneous concept taxonomies varying in availability, granularity, and semantics across cohorts. Consequently, adapting Foundation Vision-Language Models (FVLMs) demands costly label engineering and repeated post-training. Existing intervention mechanisms remain rigidly tied to predefined concepts, lacking adaptability and hindering scalable dermatology CAD deployment. To address these bottlenecks, we propose UniCon, an open-linguistic unified concept learning framework for multimodal interpretable vision-language diagnosis. UniCon resolves these challenges through three contributions: (1) A shared semantic representation space via a unified concept prototype codebook, seamlessly coordinating heterogeneous concept systems across modalities without dataset-specific retraining. (2) Open-linguistic based multi-faceted semantic specifications to overcome sparse textual label limitations, improving boundary sensitivity in uncertain clinical contexts. (3) A robust, cross-site adjustable intervention interface powered by reliability-gated bottleneck aggregation, enabling consistent reasoning and transferable clinician corrections. Extensive experiments demonstrate that beyond securing top-tier diagnostic accuracy, UniCon successfully bridges disparate clinical taxonomies, unlocking unprecedented cross-site intervention capabilities. Code is available at https://github.com/wuchengyu123/UniCon.