mlsanityData to 7 Oct 2026

Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversight

arXiv 2508.21777 ↗First posted 29 Aug 2025cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Introduction: Large language models (LLM) have shown great potential in clinical decision support. GPT-5 is a novel LLM system that has been specifically marketed towards oncology use. Methods: Performance was assessed using two complementary benchmarks: (i) the ACR Radiation Oncology In-Training Examination (TXIT, 2021), comprising 300 multiple-choice items, and (ii) a curated set of 60 authentic radiation oncologic vignettes representing diverse disease sites and treatment indications. For the vignette evaluation, GPT-5 was instructed to generate concise therapeutic plans. Four board-certified radiation oncologists rated correctness, comprehensiveness, and hallucinations. Inter-rater reliability was quantified using Fleiss' kappa. Results: On the TXIT benchmark, GPT-5 achieved a mean accuracy of 92.8%, outperforming GPT-4 (78.8%) and GPT-3.5 (62.1%). Domain-specific gains were most pronounced in Dose and Diagnosis. In the vignette evaluation, GPT-5's treatment recommendations were rated highly for correctness (mean 3.24/4, 95% CI: 3.11-3.38) and comprehensiveness (3.59/4, 95% CI: 3.49-3.69). Hallucinations were rare with no case reaching majority consensus for their presence. Inter-rater agreement was low (Fleiss' kappa 0.083 for correctness), reflecting inherent variability in clinical judgment. Errors clustered in complex scenarios requiring precise trial knowledge or detailed clinical adaptation. Discussion: GPT-5 clearly outperformed prior model variants on the radiation oncology multiple-choice benchmark. Although GPT-5 exhibited favorable performance in generating real-world radiation oncology treatment recommendations, correctness ratings indicate room for further improvement. While hallucinations were infrequent, the presence of substantive errors underscores that GPT-5-generated recommendations require rigorous expert oversight before clinical implementation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 29 Aug 2025.

Builds on

22 of 46 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
10 more
  1. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  2. 2019Publicly Available Clinical BERT Embeddingscited by 995
  3. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  4. 2021Deep learning for brain metastasis detection and segmentation in longitudinal MRI datacited by 15
  5. 2024Automated radiotherapy treatment planning guided by GPT-4Visioncited by 12
  6. 2024Fine-Tuning a Local LLaMA-3 Large Language Model for Automated Privacy-Preserving Physician Letter Generation in Radiation Oncologycited by 8
  7. 2022Deep learning for automatic head and neck lymph node level delineation provides expert-level accuracycited by 6
  8. 2024A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled optionscited by 5
  9. 2024Exploring Large Language Models for Specialist-level Oncology Carecited by 4
  10. 2025A Self-supervised Multimodal Deep Learning Approach to Differentiate Post-radiotherapy Progression from Pseudoprogression in Glioblastomacited by 4

Built on by

Not cited yet.