mlsanityData to 14 Oct 2026

Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective

arXiv 2410.16662 ↗First posted 22 Oct 2024eess.IV cs.AI cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Accurate diagnosis of ophthalmic diseases relies heavily on the interpretation of multimodal ophthalmic images, a process often time-consuming and expertise-dependent. Visual Question Answering (VQA) presents a potential interdisciplinary solution by merging computer vision and natural language processing to comprehend and respond to queries about medical images. This review article explores the recent advancements and future prospects of VQA in ophthalmology from both theoretical and practical perspectives, aiming to provide eye care professionals with a deeper understanding and tools for leveraging the underlying models. Additionally, we discuss the promising trend of large language models (LLM) in enhancing various components of the VQA framework to adapt to multimodal ophthalmic tasks. Despite the promising outlook, ophthalmic VQA still faces several challenges, including the scarcity of annotated multimodal image datasets, the necessity of comprehensive and unified evaluation methods, and the obstacles to achieving effective real-world applications. This article highlights these challenges and clarifies future directions for advancing ophthalmic VQA with LLMs. The development of LLM-based ophthalmic VQA systems calls for collaborative efforts between medical professionals and AI experts to overcome existing obstacles and advance the diagnosis and care of eye diseases.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 81 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

34 of 81 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
22 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  3. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  4. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  5. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  6. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  7. 2021Medical Visual Question Answering: A Surveycited by 93
  8. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  9. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  10. 2020DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanationcited by 43
  11. 2022Image Difference Captioning with Pre-training and Contrastive Learningcited by 39
  12. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
  13. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  14. 2022OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semanticscited by 24
  15. 2024MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertisecited by 24
  16. 2024EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imagingcited by 18
  17. 2022Consistency-preserving Visual Question Answering in Medical Imagingcited by 10
  18. 2022Recent Progress in Transformer-based Medical Image Analysiscited by 9
  19. 2024Evaluating large language models in medical applications: a surveycited by 8
  20. 2022Lesion Guided Explainable Few Weak-shot Medical Report Generationcited by 6
  21. 2022A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answeringcited by 5
  22. 2024EyeGPT: Ophthalmic Assistant with Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Can Multimodal Large Language Models Understand OCT?cited by 3