mlsanityData to 14 Oct 2026

Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V

arXiv 2310.19061 ↗First posted 29 Oct 2023cs.CVcited by 38

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

In this paper, we critically evaluate the capabilities of the state-of-the-art multimodal large language model, i.e., GPT-4 with Vision (GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly assess GPT-4V's proficiency in answering questions paired with images using both pathology and radiology datasets from 11 modalities (e.g. Microscopy, Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver, lung, etc.). Our datasets encompass a comprehensive range of medical inquiries, including sixteen distinct question types. Throughout our evaluations, we devised textual prompts for GPT-4V, directing it to synergize visual and textual information. The experiments with accuracy score conclude that the current version of GPT-4V is not recommended for real-world diagnostics due to its unreliable and suboptimal accuracy in responding to diagnostic medical questions. In addition, we delineate seven unique facets of GPT-4V's behavior in medical VQA, highlighting its constraints within this complex arena. The complete details of our evaluation cases are accessible at https://github.com/ZhilingYan/GPT4V-Medical-Report.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 16 references that are in the map. First posted on arXiv 29 Oct 2023.

Builds on

10 of 16 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  2. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 8

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  3. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  4. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  5. 2023Towards Generalist Biomedical AIcited by 289
  6. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  7. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
  8. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 23; 2025: 8; 2026: 4.

38 papers.

Sort:

By area:

  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  2. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  3. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 70
  4. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  5. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
  6. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Studycited by 28
  7. 2024LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Tracescited by 28
  8. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  9. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 23
  10. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMscited by 23
  11. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  12. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
26 more
  1. 2024Bora: Biomedical Generalist Video Generation Modelcited by 15
  2. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  3. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-trainingcited by 14
  4. 2024Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnosticscited by 11
  5. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluationcited by 11
  6. 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insightscited by 7
  7. 2024Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysiscited by 7
  8. 2024GPT-4V Cannot Generate Radiology Reports Yetcited by 7
  9. 2023Image and Data Mining in Reticular Chemistry Using GPT-4Vcited by 6
  10. 2024Benchmarking PathCLIP for Pathology Image Analysiscited by 6
  11. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  12. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4ocited by 6
  13. 2024TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotationcited by 5
  14. 2024How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perceptioncited by 5
  15. 2024Bioinformatics and Biomedical Informatics with ChatGPT: Year One Reviewcited by 4
  16. 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosiscited by 4
  17. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  18. 2026Medical thinking with multiple imagescited by 4
  19. 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgmentscited by 2
  20. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  21. 2024The Power of Combining Data and Knowledge: GPT-4o is an Effective Interpreter of Machine Learning Models in Predicting Lymph Node Metastasis of Lung Cancercited by 1
  22. 2024Predicting Lung Cancer Patient Prognosis with Large Language Modelscited by 1
  23. 2024Leveraging Multimodal Models for Enhanced Neuroimaging Diagnostics in Alzheimer's Diseasecited by 1
  24. 2025Explainable, Multi-modal Wound Infection Classification from Images Augmented with Generated Captionscited by 1
  25. 2026Locally Private Online Quantile Regression: Estimation and Inferencecited by 0
  26. 2026Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Capabilities of GPT-5 on Multimodal Medical Reasoning
  2. 2025Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology
  3. 2023An Evaluation of GPT-4V and Gemini in Online VQA
  4. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
  5. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  6. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
  7. 2023Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies
  8. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering