mlsanityData to 7 Oct 2026

A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis

arXiv 2310.20381 ↗First posted 31 Oct 2023cs.CV cs.AIcited by 37

Not on the map. Why?

Abstract (arXiv)

This work conducts an evaluation of GPT-4V's multimodal capability for medical image analysis, with a focus on three representative tasks of radiology report generation, medical visual question answering, and medical visual grounding. For the evaluation, a set of prompts is designed for each task to induce the corresponding capability of GPT-4V to produce sufficiently good outputs. Three evaluation ways including quantitative analysis, human evaluation, and case study are employed to achieve an in-depth and extensive evaluation. Our evaluation shows that GPT-4V excels in understanding medical images and is able to generate high-quality radiology reports and effectively answer questions about medical images. Meanwhile, it is found that its performance for medical visual grounding needs to be substantially improved. In addition, we observe the discrepancy between the evaluation outcome from quantitative analysis and that from human evaluation. This discrepancy suggests the limitations of conventional metrics in assessing the performance of large language models like GPT-4V and the necessity of developing new metrics for automatic quantitative analysis.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

55 of 67 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
38 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  13. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  14. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  15. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  16. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  17. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  18. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  19. 2020X-Linear Attention Networks for Image Captioningcited by 198
  20. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  21. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  22. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  23. 2020When Radiology Report Generation Meets Knowledge Graphcited by 144
  24. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  25. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  26. 2019Clinically Accurate Chest X-Ray Report Generationcited by 138
  27. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  28. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 114
  29. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  30. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 111
  31. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  32. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  33. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 71
  34. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  35. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  36. 2021Visual Grounding with Transformerscited by 42
  37. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 42
  38. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 7; 2026: 3.

37 papers.

Sort:

By area:

  1. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  2. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  3. 2023DrugAssist: A Large Language Model for Molecule Optimizationcited by 57
  4. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  5. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 33
  6. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Contextcited by 29
  7. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  8. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 21
  9. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  10. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  11. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
  12. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Modelscited by 16
25 more
  1. 2024LLM-AD: Large Language Model based Audio Description Systemcited by 14
  2. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Groundingcited by 9
  3. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  4. 2024Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patternscited by 8
  5. 2024GPT-4V Cannot Generate Radiology Reports Yetcited by 7
  6. 2024MRScore: Evaluating Radiology Report Generation with LLM-based Reward Systemcited by 6
  7. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMscited by 6
  8. 2025Visual Question Answering on Multiple Remote Sensing Image Modalitiescited by 6
  9. 2023An Evaluation of GPT-4V and Gemini in Online VQAcited by 5
  10. 2024Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Modelcited by 5
  11. 2025S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Frameworkcited by 5
  12. 2024Bioinformatics and Biomedical Informatics with ChatGPT: Year One Reviewcited by 4
  13. 2025CartoAgent: a multimodal large language model-powered multi-agent cartographic framework for map style transfer and evaluationcited by 4
  14. 2024Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignmentcited by 3
  15. 2024MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosiscited by 3
  16. 2025Look & Mark: Leveraging Radiologist Eye Fixations and Bounding boxes in Multimodal Large Language Models for Chest X-ray Report Generationcited by 3
  17. 2024CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4Vcited by 2
  18. 2025Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Datacited by 2
  19. 2025RadReason: Radiology Report Evaluation Metric with Reasons and Sub-Scorescited by 2
  20. 2025Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agentscited by 1
  21. 2026Seeing What Matters: Lesion-Aware High-Resolution Patch Discovery and Fusion for Chest X-ray Report Generationcited by 1
  22. 2023Trustworthy Large Models in Vision: A Surveycited by 0
  23. 2023Visual AI and Linguistic Intelligence Through Steerability and Composabilitycited by 0
  24. 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasoundcited by 0
  25. 2026Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Modelscited by 0