# Exploring the Boundaries of GPT-4 in Radiology

arXiv: https://arxiv.org/abs/2310.14573 · HTML: https://mlsanity.com/paper/2310.14573/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT](https://mlsanity.com/paper/2304.08448/) (2023)
- [Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential](https://mlsanity.com/paper/2303.09038/) (2023)
- [Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task](https://mlsanity.com/paper/2304.09138/) (2023)
- [Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models](https://mlsanity.com/paper/2305.03660/) (2023)
- [Graph Enhanced Contrastive Learning for Radiology Findings Summarization](https://mlsanity.com/paper/2204.00203/) (2022)
- [Compositional Zero-Shot Domain Transfer with Text-to-Text Models](https://mlsanity.com/paper/2303.13386/) (2023)

## Built on by (most cited first, 30 of 31)

- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://mlsanity.com/paper/2410.06542/) (2024)
- [RareBench: Can LLMs Serve as Rare Diseases Specialists?](https://mlsanity.com/paper/2402.06341/) (2024)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation](https://mlsanity.com/paper/2411.05085/) (2024)
- [LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey](https://mlsanity.com/paper/2402.14558/) (2024)
- [Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification](https://mlsanity.com/paper/2403.04024/) (2024)
- [SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignment](https://mlsanity.com/paper/2410.14676/) (2024)
- [GPT-4V Cannot Generate Radiology Reports Yet](https://mlsanity.com/paper/2407.12176/) (2024)
- [SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval](https://mlsanity.com/paper/2401.02369/) (2024)
- [Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation](https://mlsanity.com/paper/2407.01948/) (2024)
- [Enhancing disease detection in radiology reports through fine-tuning lightweight LLM on weak labels](https://mlsanity.com/paper/2409.16563/) (2024)
- [Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reports](https://mlsanity.com/paper/2402.12298/) (2024)
- [RAD-PHI2: Instruction Tuning PHI-2 for Radiology](https://mlsanity.com/paper/2403.09725/) (2024)
- [A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification](https://mlsanity.com/paper/2401.13887/) (2024)
- [No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size](https://mlsanity.com/paper/2408.01444/) (2024)
- [Large Language Models for Healthcare Text Classification: A Systematic Review](https://mlsanity.com/paper/2503.01159/) (2025)
- [Cancer Type, Stage and Prognosis Assessment from Pathology Reports using LLMs](https://mlsanity.com/paper/2503.01194/) (2025)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [Leveraging Prompt-Learning for Structured Information Extraction from Crohn's Disease Radiology Reports in a Low-Resource Language](https://mlsanity.com/paper/2405.01682/) (2024)
- [Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings](https://mlsanity.com/paper/2406.16611/) (2024)
- [Can Rule-Based Insights Enhance LLMs for Radiology Report Classification? Introducing the RadPrompt Methodology](https://mlsanity.com/paper/2408.04121/) (2024)
- [HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation](https://mlsanity.com/paper/2409.13038/) (2024)
- [Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology](https://mlsanity.com/paper/2509.25559/) (2025)
- [RadBARTsum: Domain Specific Adaption of Denoising Sequence-to-Sequence Models for Abstractive Radiology Report Summarization](https://mlsanity.com/paper/2406.03062/) (2024)
- [Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systems](https://mlsanity.com/paper/2402.00234/) (2024)
- [Inferring Pluggable Types with Machine Learning](https://mlsanity.com/paper/2406.15676/) (2024)
- [Building Models of Neurological Language](https://mlsanity.com/paper/2506.06208/) (2025)
- [CR3G: Causal Reasoning for Patient-Centric Explanations in Radiology Report Generation](https://mlsanity.com/paper/2512.11830/) (2025)

## Abstract (arXiv)

The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-specific models. Exploring various prompting strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology models. With zero-shot prompting, GPT-4 already obtains substantial gains (≈ 10% absolute improvement) over radiology models in temporal sentence similarity classification (accuracy) and natural language inference (F1). For tasks that require learning dataset-specific style or schema (e.g. findings summarisation), GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive error analysis with a board-certified radiologist shows GPT-4 has a sufficient level of radiology knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT-4 outputs are found to be overall comparable with existing manually-written impressions.