Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Artificial Intelligence models have demonstrated significant success in diagnosing skin diseases, including cancer, showing the potential to assist clinicians in their analysis. However, the interpretability of model predictions must be significantly improved before they can be used in practice. To this end, we explore the combination of two promising approaches: Multimodal Large Language Models (MLLMs) and quantitative attribute usage. MLLMs offer a potential avenue for increased interpretability, providing reasoning for diagnosis in natural language through an interactive format. Separately, a number of quantitative attributes that are related to lesion appearance (e.g., lesion area) have recently been found predictive of malignancy with high accuracy. Predictions grounded as a function of such concepts have the potential for improved interpretability. We provide evidence that MLLM embedding spaces can be grounded in such attributes, through fine-tuning to predict their values from images. Concretely, we evaluate this grounding in the embedding space through an attribute-specific content-based image retrieval case study using the SLICE-3D dataset.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 24 references that are in the map. First posted on arXiv 27 Aug 2025.
Builds on
15 of 24 references.
Closest ancestors · 2
Same direction, newest first
- 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks9 May
- 2025Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology19 Mar
Other references · 13
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Qwen2 Technical Reportcited by 2,120
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
- 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 236
- 2024A Multimodal Vision Foundation Model for Clinical Dermatologycited by 50
- 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 46
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Towards Concept-based Interpretability of Skin Lesion Diagnosis using Vision-Language Models
- 2025On the Importance of Text Preprocessing for Multimodal Representation Learning and Pathology Report Generation
- 2025Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis
- 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
- 2025ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images
- 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
- 2025VAP-Diffusion: Enriching Descriptions with MLLMs for Enhanced Medical Image Generation
- 2025MedVision: Benchmarking Quantitative Medical Image Analysis