mlsanityData to 7 Oct 2026

Retina-RAG: Retrieval-Augmented Vision-Language Modeling for Joint Retinal Diagnosis and Clinical Report Generation

arXiv 2605.06173 ↗First posted 7 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Diabetic Retinopathy (DR) is a leading cause of preventable blindness among working-age adults worldwide, yet most automated screening systems are limited to image-level classification and lack clinically structured reporting. We propose Retina-RAG, a low-cost modular framework that jointly performs DR severity grading, macular edema (ME) detection, and report generation. The architecture decouples a high-performance retinal classifier and a parameter-efficient vision-language model (Qwen2.5-VL-7B-Instruct) adapted via Low-Rank Adaptation (LoRA), enabling flexible component integration. A retrieval-augmented generation (RAG) module injects curated ophthalmic knowledge together with structured classifier outputs at inference time to improve diagnostic consistency and reduce hallucinations. Retina-RAG achieves an F1-score of 0.731 for DR grading and 0.948 for ME detection, substantially outperforming zero-shot Qwen (0.096, 0.732) and MMed-RAG (0.541, 0.641) on a retinal disease detection dataset with captions. For report generation, Retina-RAG attains ROUGE-L 0.438 and SBERT similarity 0.884, exceeding all baselines. The full framework operates on a single consumer-grade GPU, demonstrating that clinically structured retinal AI can be achieved with modest computational resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 38 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

18 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models16 Oct
  2. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models6 Jul
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar
  4. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging5 Jan
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 13

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
1 more
  1. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Pixels to Explanations: Interpretable Diabetic Retinopathy Grading with CNN-Transformer Ensembles, Visual Explainability and Vision-Language Models
  2. 2024CLIP-DR: Textual Knowledge-Guided Diabetic Retinopathy Grading with Ranking-aware Prompting
  3. 2026FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs
  4. 2025Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Models
  5. 2026Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening
  6. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
  7. 2025RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
  8. 2024RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports