# Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images

arXiv: https://arxiv.org/abs/2510.21839 · HTML: https://mlsanity.com/paper/2510.21839/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis](https://mlsanity.com/paper/2307.01981/) (2023)
- [HealthPrompt: A Zero-shot Learning Paradigm for Clinical Natural Language Processing](https://mlsanity.com/paper/2203.05061/) (2022)
- [Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports](https://mlsanity.com/paper/2110.03094/) (2021)
- [Pneumonia Detection on chest X-ray images Using Ensemble of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2312.07965/) (2023)
- [GPT-4V Cannot Generate Radiology Reports Yet](https://mlsanity.com/paper/2407.12176/) (2024)
- [PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification](https://mlsanity.com/paper/2404.08915/) (2024)

## Abstract (arXiv)

In this study, we evaluate the ability of OpenAI's gpt-4o model to classify chest X-ray images as either NORMAL or PNEUMONIA in a zero-shot setting, without any prior fine-tuning. A balanced test set of 400 images (200 from each class) was used to assess performance across four distinct prompt designs, ranging from minimal instructions to detailed, reasoning-based prompts. The results indicate that concise, feature-focused prompts achieved the highest classification accuracy of 74%, whereas reasoning-oriented prompts resulted in lower performance. These findings highlight that while ChatGPT exhibits emerging potential for medical image interpretation, its diagnostic reliability remains limited. Continued advances in visual reasoning and domain-specific adaptation are required before such models can be safely applied in clinical practice.