# Glaucoma Detection and Structured OCT Report Generation via a Fine-tuned Multimodal Large Language Model

arXiv: https://arxiv.org/abs/2510.02403 · HTML: https://mlsanity.com/paper/2510.02403/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 24 of 24)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [Exploring large scale public medical image datasets](https://mlsanity.com/paper/1907.12720/) (2019)
- [Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism](https://mlsanity.com/paper/2311.01041/) (2023)
- [Beyond Text: Frozen Large Language Models in Visual Signal Comprehension](https://mlsanity.com/paper/2403.07874/) (2024)
- [RadEval: A framework for radiology text evaluation](https://mlsanity.com/paper/2509.18030/) (2025)
- [Explainable artificial intelligence (XAI): from inherent explainability to large language models](https://mlsanity.com/paper/2501.09967/) (2025)
- [Improving Multi-modal Large Language Model through Boosting Vision Capabilities](https://mlsanity.com/paper/2410.13733/) (2024)
- [Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalities](https://mlsanity.com/paper/2508.07031/) (2025)
- [The current status of large language models in summarizing radiology report impressions](https://mlsanity.com/paper/2406.02134/) (2024)
- [Explainable AI for clinical risk prediction: a survey of concepts, methods, and modalities](https://mlsanity.com/paper/2308.08407/) (2023)
- [Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration](https://mlsanity.com/paper/2505.06898/) (2025)
- [DeepEyeNet: Generating Medical Report for Retinal Images](https://mlsanity.com/paper/2509.12534/) (2025)

## Built on by (most cited first, 2 of 2)

- [Fair-Eye Net: A Fair, Trustworthy, Multimodal Integrated Glaucoma Full Chain AI System](https://mlsanity.com/paper/2601.18464/) (2026)
- [Representation learning from OCT images](https://mlsanity.com/paper/2605.02589/) (2026)

## Abstract (arXiv)

Objective: To develop an explainable multimodal large language model (MM-LLM) that (1) screens optic nerve head (ONH) OCT circle scans for quality and (2) generates structured clinical reports that include glaucoma diagnosis and sector-wise retinal nerve fiber layer (RNFL) thinning assessments. Design: Retrospective cohort study of 1,310 subjects contributing 43,849 Spectralis ONH OCT circle scans (1,331 glaucomatous and 867 healthy eyes) from the DIGS and ADAGES cohorts. Methods: A MM-LLM (Llama 3.2 Vision-Instruct model) was fine-tuned to generate clinical descriptions of OCT imaging data. Training data included paired OCT images and automatically generated, structured clinical reports that described global and sectoral RNFL thinning. Poor-quality scans were labeled as unusable and paired with a fixed refusal statement. The model was evaluated on a held-out test set for three tasks: quality assessment, glaucoma detection, and RNFL thinning classification across seven anatomical sectors. Evaluation metrics included accuracy, sensitivity, specificity, precision, and F1-score. Model description quality was also evaluated using standard text evaluation metrics. Results: The model achieved 0.90 accuracy and 0.98 specificity for quality triage. For glaucoma detection, accuracy was 0.86 (sensitivity 0.91, specificity 0.73, F1-score 0.91). RNFL thinning prediction accuracy ranged from 0.83 to 0.94, with highest performance in global and temporal sectors. Text generation scores showed strong alignment with reference reports (BLEU: 0.82; ROUGE-1: 0.94; ROUGE-2: 0.87; ROUGE-L: 0.92; BERTScore-F1: 0.99). Conclusions: The fine-tuned MM-LLM generated accurate clinical descriptions based on OCT imaging. The model achieved high accuracy in identifying image quality issues and detecting glaucoma. The model also provided sectoral descriptions of RNFL thinning to help support clinical OCT evaluation.