mlsanityData to 7 Oct 2026

Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

arXiv 2310.09909 ↗First posted 15 Oct 2023cs.CV cs.CLcited by 77

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Driven by the large foundation models, the development of artificial intelligence has witnessed tremendous progress lately, leading to a surge of general interest from the public. In this study, we aim to assess the performance of OpenAI's newest model, GPT-4V(ision), specifically in the realm of multimodal medical diagnosis. Our evaluation encompasses 17 human body systems, including Central Nervous System, Head and Neck, Cardiac, Chest, Hematology, Hepatobiliary, Gastrointestinal, Urogenital, Gynecology, Obstetrics, Breast, Musculoskeletal, Spine, Vascular, Oncology, Trauma, Pediatrics, with images taken from 8 modalities used in daily clinic routine, e.g., X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Digital Subtraction Angiography (DSA), Mammography, Ultrasound, and Pathology. We probe the GPT-4V's ability on multiple clinical tasks with or without patent history provided, including imaging modality and anatomy recognition, disease diagnosis, report generation, disease localisation. Our observation shows that, while GPT-4V demonstrates proficiency in distinguishing between medical image modalities and anatomy, it faces significant challenges in disease diagnosis and generating comprehensive reports. These findings underscore that while large multimodal models have made significant advancements in computer vision and natural language processing, it remains far from being used to effectively support real-world medical applications and clinical decision-making. All images used in this report can be found in https://github.com/chaoyi-wu/GPT-4V_Medical_Evaluation.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 9 of its 27 references that are in the map. First posted on arXiv 15 Oct 2023.

Builds on

9 of 27 references.

References · 9

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  3. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 507
  4. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  5. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  6. 2023Towards Generalist Biomedical AIcited by 286
  7. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  8. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  9. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 41; 2025: 14; 2026: 4.

77 papers.

Sort:

By area:

  1. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  2. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  3. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  4. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  5. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  6. 2024Can ChatGPT Detect DeepFakes? A Study of Using Multimodal Large Language Models for Media Forensicscited by 68
  7. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  8. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  9. 2024Dense Connector for MLLMscited by 48
  10. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  11. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  12. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
38 more
  1. 2023Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Leadcited by 40
  2. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  3. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  4. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
  5. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicinecited by 28
  6. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
  7. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22
  8. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 21
  9. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  10. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  11. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 19
  12. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  13. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalitiescited by 18
  14. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 18
  15. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
  16. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Modelscited by 16
  17. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 16
  18. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  19. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  20. 2024LLM-AD: Large Language Model based Audio Description Systemcited by 14
  21. 2024ChatGPT Meets Iris Biometricscited by 14
  22. 2023NERIF: GPT-4V for Automatic Scoring of Drawn Modelscited by 13
  23. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12
  24. 2024By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Promptingcited by 12
  25. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  26. 2024AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agentscited by 11
  27. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluationcited by 11
  28. 2023Dietary Assessment with Multimodal ChatGPT: A Systematic Analysiscited by 10
  29. 2025End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoningcited by 10
  30. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Modelscited by 9
  31. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  32. 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insightscited by 7
  33. 2024Embedded Visual Prompt Tuningcited by 7
  34. 2024GPT-4V Cannot Generate Radiology Reports Yetcited by 7
  35. 2023GPT-4V(ision) as A Social Media Analysis Enginecited by 6
  36. 2023Image and Data Mining in Reticular Chemistry Using GPT-4Vcited by 6
  37. 2024Harnessing GPT-4V(ision) for Insurance: A Preliminary Explorationcited by 5
  38. 2024Low-Rank Knowledge Decomposition for Medical Foundation Modelscited by 5

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GPT-4V(ision) Unsuitable for Clinical Care and Education: A Clinician-Evaluated Assessment
  2. 2024Evaluating GPT-4 with Vision on Detection of Radiological Findings on Chest Radiographs
  3. 2024How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses
  4. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V
  5. 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
  6. 2023Artificial General Intelligence for Medical Imaging Analysis
  7. 2025Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology
  8. 2025MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images