mlsanityData to 7 Oct 2026

Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies

arXiv 2312.04344 ↗First posted 7 Dec 2023cs.CL cs.AI cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

OpenAI's latest large vision-language model (LVLM), GPT-4V(ision), has piqued considerable interest for its potential in medical applications. Despite its promise, recent studies and internal reviews highlight its underperformance in specialized medical tasks. This paper explores the boundary of GPT-4V's capabilities in medicine, particularly in processing complex imaging data from endoscopies, CT scans, and MRIs etc. Leveraging open-source datasets, we assessed its foundational competencies, identifying substantial areas for enhancement. Our research emphasizes prompt engineering, an often-underutilized strategy for improving AI responsiveness. Through iterative testing, we refined the model's prompts, significantly improving its interpretative accuracy and relevance in medical imaging. From our comprehensive evaluations, we distilled 10 effective prompt engineering techniques, each fortifying GPT-4V's medical acumen. These methodical enhancements facilitate more reliable, precise, and clinically valuable insights from GPT-4V, advancing its operability in critical healthcare environments. Our findings are pivotal for those employing AI in medicine, providing clear, actionable guidance on harnessing GPT-4V's full diagnostic potential.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 14 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

7 of 14 references.

References · 7

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 347
  4. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  5. 2020m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networkscited by 25
  6. 2023Multimodal Foundation Models Exploit Text to Make Medical Image Predictionscited by 19
  7. 2022AutoPET Challenge 2022: Automatic Segmentation of Whole-body Tumor Lesion Based on Deep Learning and FDG PET/CTcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Explainable, Multi-modal Wound Infection Classification from Images Augmented with Generated Captionscited by 1
  2. 2025Anatomy-Aware Text-Visual Fusion with Dual-Perspective Prompts for Fine-Grained Lumbar Spine Segmentationcited by 1
  3. 2024UMass-BioNLP at MEDIQA-M3G 2024: DermPrompt -- A Systematic Exploration of Prompt Engineering with GPT-4V for Dermatological Diagnosiscited by 0
  4. 2026Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Holistic Evaluation of GPT-4V for Biomedical Imaging
  2. 2025Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks
  3. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V
  4. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
  5. 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
  6. 2023Artificial General Intelligence for Medical Imaging Analysis
  7. 2024GPT-4V Cannot Generate Radiology Reports Yet
  8. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks