mlsanityData to 14 Oct 2026

Holistic Evaluation of GPT-4V for Biomedical Imaging

arXiv 2312.05256 ↗First posted 10 Nov 2023eess.IV cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a large-scale evaluation probing GPT-4V's capabilities and limitations for biomedical image analysis. GPT-4V represents a breakthrough in artificial general intelligence (AGI) for computer vision, with applications in the biomedical domain. We assess GPT-4V's performance across 16 medical imaging categories, including radiology, oncology, ophthalmology, pathology, and more. Tasks include modality recognition, anatomy localization, disease diagnosis, report generation, and lesion detection. The extensive experiments provide insights into GPT-4V's strengths and weaknesses. Results show GPT-4V's proficiency in modality and anatomy recognition but difficulty with disease diagnosis and localization. GPT-4V excels at diagnostic report generation, indicating strong image captioning skills. While promising for biomedical imaging AI, GPT-4V requires further enhancement and validation before clinical deployment. We emphasize responsible development and testing for trustworthy integration of biomedical AGI. This rigorous evaluation of GPT-4V on diverse medical images advances understanding of multimodal large language models (LLMs) and guides future work toward impactful healthcare applications.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 91 of its 234 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

91 of 234 references.

References · 91

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  12. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  13. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  14. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  15. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  16. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  17. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  18. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  19. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  20. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  21. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  22. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  23. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  24. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  25. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  26. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 131
  27. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 130
  28. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  29. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  30. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  31. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  32. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  33. 2023Cumulative Reasoning with Large Language Modelscited by 94
  34. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
  35. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  36. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  37. 2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 80
  38. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 13; 2025: 5.

19 papers.

Sort:

By area:

  1. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  2. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  3. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  4. 2024Large Language Models for Manufacturingcited by 24
  5. 2024Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysiscited by 21
  6. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  7. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 19
  8. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15
  9. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  10. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 11
  11. 2024Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarioscited by 10
  12. 2024RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-trainingcited by 8
7 more
  1. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approachcited by 8
  2. 2024GPT-4V Cannot Generate Radiology Reports Yetcited by 7
  3. 2024Beyond Accuracy: Ensuring Correct Predictions With Correct Rationalescited by 6
  4. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4ocited by 6
  5. 2024Bioinformatics and Biomedical Informatics with ChatGPT: Year One Reviewcited by 4
  6. 2025Large Language Models for Bioinformaticscited by 3
  7. 2025Fusion-Augmented Large Language Models: Boosting Diagnostic Trustworthiness via Model Consensuscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies
  2. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V
  3. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
  4. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
  5. 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
  6. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis
  7. 2025MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images
  8. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks