mlsanityData to 7 Oct 2026

OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue

arXiv 2306.12174 ↗First posted 21 Jun 2023cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large multimodal language models (LMMs) have achieved significant success in general domains. However, due to the significant differences between medical images and text and general web content, the performance of LMMs in medical scenarios is limited. In ophthalmology, clinical diagnosis relies on multiple modalities of medical images, but unfortunately, multimodal ophthalmic large language models have not been explored to date. In this paper, we study and construct an ophthalmic large multimodal model. Firstly, we use fundus images as an entry point to build a disease assessment and diagnosis pipeline to achieve common ophthalmic disease diagnosis and lesion segmentation. Then, we establish a new ophthalmic multimodal instruction-following and dialogue fine-tuning dataset based on disease-related knowledge data and publicly available real-world medical dialogue. We introduce visual ability into the large language model to complete the ophthalmic large language and vision assistant (OphGLM). Our experimental results demonstrate that the OphGLM model performs exceptionally well, and it has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be made publicly available at https://github.com/ML-AILab/OphGLM.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 25 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

16 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  2. 2022Self-supervised vision-language pretraining for Medical visual question answering24 Nov

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  5. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  6. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  8. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  9. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  10. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  11. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  12. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
2 more
  1. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  2. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 15; 2025: 10; 2026: 2.

34 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  3. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  4. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  5. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  6. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  7. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  8. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  9. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  10. 2024GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaborationcited by 33
  11. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  12. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
22 more
  1. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  2. 2023From Beginner to Expert: Modeling Medical Knowledge into General LLMscited by 14
  3. 2024Unearthing Large Scale Domain-Specific Knowledge from Public Corporacited by 12
  4. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8
  5. 2024PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulationcited by 8
  6. 2025FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Frameworkcited by 8
  7. 2024D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictionscited by 7
  8. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decodingcited by 6
  9. 2024Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Promptingcited by 5
  10. 2023CephGPT-4: An Interactive Multimodal Cephalometric Measurement and Diagnostic System with Visual Large Language Modelcited by 4
  11. 2023Evaluating multiple large language models in pediatric ophthalmologycited by 4
  12. 2024Enhance the Robustness of Text-Centric Multimodal Alignmentscited by 4
  13. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  14. 2025A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Modelscited by 3
  15. 2025Generative AI for Healthcare: Fundamentals, Challenges, and Perspectivescited by 3
  16. 2024Evaluation of Language Models in the Medical Context Under Resource-Constrained Settingscited by 1
  17. 2024A Survey for Large Language Models in Biomedicinecited by 1
  18. 2024A Medical Multimodal Large Language Model for Pediatric Pneumoniacited by 1
  19. 2024Visual Question Answering in Ophthalmology: A Progressive and Practical Perspectivecited by 1
  20. 2026LLaVA-LE: Large Language-and-Vision Assistant for Lunar Explorationcited by 1
  21. 2025RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Modelcited by 0
  22. 2026Improving Medical Calculation of LLMs with Embedded Codingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models
  2. 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
  3. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models
  4. 2026VOLMO: Versatile and Open Large Models for Ophthalmology
  5. 2025EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model
  6. 2025A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Images
  7. 2025Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items
  8. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding