mlsanityData to 7 Oct 2026

Exploring the Boundaries of GPT-4 in Radiology

arXiv 2310.14573 ↗First posted 23 Oct 2023cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-specific models. Exploring various prompting strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology models. With zero-shot prompting, GPT-4 already obtains substantial gains (≈ 10% absolute improvement) over radiology models in temporal sentence similarity classification (accuracy) and natural language inference (F1). For tasks that require learning dataset-specific style or schema (e.g. findings summarisation), GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive error analysis with a board-certified radiologist shows GPT-4 has a sufficient level of radiology knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT-4 outputs are found to be overall comparable with existing manually-written impressions.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

23 of 42 references.

References · 23

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  11. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  12. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
11 more
  1. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  2. 2019Entity, Relation, and Event Extraction with Contextualized Span Representationscited by 261
  3. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  4. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  5. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  6. 2023An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPTcited by 57
  7. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39
  8. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Taskcited by 32
  9. 2023Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT modelscited by 27
  10. 2022Graph Enhanced Contrastive Learning for Radiology Findings Summarizationcited by 24
  11. 2023Compositional Zero-Shot Domain Transfer with Text-to-Text Modelscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 24; 2025: 7.

31 papers.

Sort:

By area:

  1. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  2. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  3. 2024RareBench: Can LLMs Serve as Rare Diseases Specialists?cited by 45
  4. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 45
  5. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 41
  6. 2024PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generationcited by 35
  7. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  8. 2024Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classificationcited by 8
  9. 2024SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignmentcited by 8
  10. 2024GPT-4V Cannot Generate Radiology Reports Yetcited by 7
  11. 2024SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrievalcited by 5
  12. 2024Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representationcited by 5
19 more
  1. 2024Enhancing disease detection in radiology reports through fine-tuning lightweight LLM on weak labelscited by 5
  2. 2024Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reportscited by 4
  3. 2024RAD-PHI2: Instruction Tuning PHI-2 for Radiologycited by 4
  4. 2024A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classificationcited by 3
  5. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  6. 2025Large Language Models for Healthcare Text Classification: A Systematic Reviewcited by 3
  7. 2025Cancer Type, Stage and Prognosis Assessment from Pathology Reports using LLMscited by 3
  8. 2025Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Reviewcited by 3
  9. 2024Leveraging Prompt-Learning for Structured Information Extraction from Crohn's Disease Radiology Reports in a Low-Resource Languagecited by 2
  10. 2024Can Rule-Based Insights Enhance LLMs for Radiology Report Classification? Introducing the RadPrompt Methodologycited by 2
  11. 2024HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generationcited by 2
  12. 2025Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiologycited by 2
  13. 2024RadBARTsum: Domain Specific Adaption of Denoising Sequence-to-Sequence Models for Abstractive Radiology Report Summarizationcited by 1
  14. 2024Evaluation of Language Models in the Medical Context Under Resource-Constrained Settingscited by 1
  15. 2024Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systemscited by 0
  16. 2024Inferring Pluggable Types with Machine Learningcited by 0
  17. 2025Building Models of Neurological Languagecited by 0
  18. 2025CR3G: Causal Reasoning for Patient-Centric Explanations in Radiology Report Generationcited by 0
  19. 2025brat: Aligned Multi-View Embeddings for Brain MRI Analysiscited by 0