# Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Models

arXiv: https://arxiv.org/abs/2503.02917 · HTML: https://mlsanity.com/paper/2503.02917/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Prompt Distribution Learning](https://mlsanity.com/paper/2205.03340/) (2022)
- [Applications of Deep Learning in Fundus Images: A Review](https://mlsanity.com/paper/2101.09864/) (2021)
- [Disc-aware Ensemble Network for Glaucoma Screening from Fundus Image](https://mlsanity.com/paper/1805.07549/) (2018)
- [Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition](https://mlsanity.com/paper/2104.11057/) (2021)
- [DRvertGRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images](https://mlsanity.com/paper/1910.11777/) (2019)
- [Automatic detection of rare pathologies in fundus photographs using few-shot learning](https://mlsanity.com/paper/1907.09449/) (2019)
- [Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognition](https://mlsanity.com/paper/2111.08913/) (2021)

## Built on by (most cited first, 4 of 4)

- [WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classification](https://mlsanity.com/paper/2509.17740/) (2025)
- [Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2605.07140/) (2026)
- [NeRD: Neuro-Symbolic Rule Distillation for Efficient Ontology-Grounded Chain-of-Thought in Medical Image Diagnosis](https://mlsanity.com/paper/2606.15617/) (2026)
- [VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences](https://mlsanity.com/paper/2609.00921/) (2026)

## Abstract (arXiv)

Recent advancements in deep learning have shown significant potential for classifying retinal diseases using color fundus images. However, existing works predominantly rely exclusively on image data, lack interpretability in their diagnostic decisions, and treat medical professionals primarily as annotators for ground truth labeling. To fill this gap, we implement two key strategies: extracting interpretable concepts of retinal diseases using the knowledge base of GPT models and incorporating these concepts as a language component in prompt-learning to train vision-language (VL) models with both fundus images and their associated concepts. Our method not only improves retinal disease classification but also enriches few-shot and zero-shot detection (novel disease detection), while offering the added benefit of concept-based model interpretability. Our extensive evaluation across two diverse retinal fundus image datasets illustrates substantial performance gains in VL-model based few-shot methodologies through our concept integration approach, demonstrating an average improvement of approximately 5.8% and 2.7% mean average precision for 16-shot learning and zero-shot (novel class) detection respectively. Our method marks a pivotal step towards interpretable and efficient retinal disease recognition for real-world clinical applications.