# Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies

arXiv: https://arxiv.org/abs/2312.04344 · HTML: https://mlsanity.com/paper/2312.04344/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 7 of 7)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023)
- [m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks](https://mlsanity.com/paper/2008.10134/) (2020)
- [Multimodal Foundation Models Exploit Text to Make Medical Image Predictions](https://mlsanity.com/paper/2311.05591/) (2023)
- [AutoPET Challenge 2022: Automatic Segmentation of Whole-body Tumor Lesion Based on Deep Learning and FDG PET/CT](https://mlsanity.com/paper/2209.01212/) (2022)

## Built on by (most cited first, 4 of 4)

- [Explainable, Multi-modal Wound Infection Classification from Images Augmented with Generated Captions](https://mlsanity.com/paper/2502.20277/) (2025)
- [Anatomy-Aware Text-Visual Fusion with Dual-Perspective Prompts for Fine-Grained Lumbar Spine Segmentation](https://mlsanity.com/paper/2504.03476/) (2025)
- [UMass-BioNLP at MEDIQA-M3G 2024: DermPrompt -- A Systematic Exploration of Prompt Engineering with GPT-4V for Dermatological Diagnosis](https://mlsanity.com/paper/2404.17749/) (2024)
- [Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models](https://mlsanity.com/paper/2607.21998/) (2026)

## Abstract (arXiv)

OpenAI's latest large vision-language model (LVLM), GPT-4V(ision), has piqued considerable interest for its potential in medical applications. Despite its promise, recent studies and internal reviews highlight its underperformance in specialized medical tasks. This paper explores the boundary of GPT-4V's capabilities in medicine, particularly in processing complex imaging data from endoscopies, CT scans, and MRIs etc. Leveraging open-source datasets, we assessed its foundational competencies, identifying substantial areas for enhancement. Our research emphasizes prompt engineering, an often-underutilized strategy for improving AI responsiveness. Through iterative testing, we refined the model's prompts, significantly improving its interpretative accuracy and relevance in medical imaging. From our comprehensive evaluations, we distilled 10 effective prompt engineering techniques, each fortifying GPT-4V's medical acumen. These methodical enhancements facilitate more reliable, precise, and clinically valuable insights from GPT-4V, advancing its operability in critical healthcare environments. Our findings are pivotal for those employing AI in medicine, providing clear, actionable guidance on harnessing GPT-4V's full diagnostic potential.