# RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models

arXiv: https://arxiv.org/abs/2503.03987 · HTML: https://mlsanity.com/paper/2503.03987/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](https://mlsanity.com/paper/2204.08790/) (2022)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Evaluation of Retinal Image Quality Assessment Networks in Different Color-spaces](https://mlsanity.com/paper/1907.05345/) (2019)
- [A Benchmark of Ocular Disease Intelligent Recognition: One Shot for Multi-disease Detection](https://mlsanity.com/paper/2102.07978/) (2021)
- [CUNSB-RFIE: Context-aware Unpaired Neural Schr"odinger Bridge in Retinal Fundus Image Enhancement](https://mlsanity.com/paper/2409.10966/) (2024)
- [nnMobileNet: Rethinking CNN for Retinopathy Research](https://mlsanity.com/paper/2306.01289/) (2023)
- [Context-Aware Optimal Transport Learning for Retinal Fundus Image Enhancement](https://mlsanity.com/paper/2409.07862/) (2024)
- [RBAD: A Dataset and Benchmark for Retinal Vessels Branching Angle Detection](https://mlsanity.com/paper/2407.12271/) (2024)
- [EyeBench: A Call for More Rigorous Evaluation of Retinal Image Enhancement](https://mlsanity.com/paper/2502.14260/) (2025)

## Built on by (most cited first, 10 of 10)

- [LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding](https://mlsanity.com/paper/2508.01617/) (2025)
- [Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning](https://mlsanity.com/paper/2508.16129/) (2025)
- [Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation](https://mlsanity.com/paper/2503.08906/) (2025)
- [A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models](https://mlsanity.com/paper/2508.03734/) (2025)
- [EZBlender: Efficient 3D Editing with Plan-and-ReAct Agent](https://mlsanity.com/paper/2601.07143/) (2026)
- [Fundus-R1: Training a Fundus-Reading MLLM with Knowledge-Aware Reasoning on Public Data](https://mlsanity.com/paper/2604.08322/) (2026)
- [nnMobileNet++: Towards Efficient Hybrid Networks for Retinal Image Analysis](https://mlsanity.com/paper/2512.01273/) (2025)
- [RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model](https://mlsanity.com/paper/2510.04281/) (2025)
- [VAOT: Vessel-Aware Optimal Transport for Retinal Fundus Enhancement](https://mlsanity.com/paper/2511.18763/) (2025)
- [Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning](https://mlsanity.com/paper/2605.27960/) (2026)

## Abstract (arXiv)

Recently, Multimodal Large Language Models (MLLMs) have gained significant attention for their remarkable ability to process and analyze non-textual data, such as images, videos, and audio. Notably, several adaptations of general-domain MLLMs to the medical field have been explored, including LLaVA-Med. However, these medical adaptations remain insufficiently advanced in understanding and interpreting retinal images. In contrast, medical experts emphasize the importance of quantitative analyses for disease detection and interpretation. This underscores a gap between general-domain and medical-domain MLLMs: while general-domain MLLMs excel in broad applications, they lack the specialized knowledge necessary for precise diagnostic and interpretative tasks in the medical field. To address these challenges, we introduce textitRetinalGPT, a multimodal conversational assistant for clinically preferred quantitative analysis of retinal images. Specifically, we achieve this by compiling a large retinal image dataset, developing a novel data pipeline, and employing customized visual instruction tuning to enhance both retinal analysis and enrich medical knowledge. In particular, RetinalGPT outperforms MLLM in the generic domain by a large margin in the diagnosis of retinal diseases in 8 benchmark retinal datasets. Beyond disease diagnosis, RetinalGPT features quantitative analyses and lesion localization, representing a pioneering step in leveraging LLMs for an interpretable and end-to-end clinical research framework. The code is available at https://github.com/Retinal-Research/RetinalGPT