mlsanityData to 7 Oct 2026

Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology

arXiv 2304.11957 ↗First posted 24 Apr 2023physics.med-ph cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The potential of large language models in medicine for education and decision making purposes has been demonstrated as they achieve decent scores on medical exams such as the United States Medical Licensing Exam (USMLE) and the MedQA exam. In this work, we evaluate the performance of ChatGPT-4 in the specialized field of radiation oncology using the 38th American College of Radiology (ACR) radiation oncology in-training (TXIT) exam and the 2022 Red Journal Gray Zone cases. For the TXIT exam, ChatGPT-3.5 and ChatGPT-4 have achieved the scores of 63.65% and 74.57%, respectively, highlighting the advantage of the latest ChatGPT-4 model. Based on the TXIT exam, ChatGPT-4's strong and weak areas in radiation oncology are identified to some extent. Specifically, ChatGPT-4 demonstrates better knowledge of statistics, CNS & eye, pediatrics, biology, and physics than knowledge of bone & soft tissue and gynecology, as per the ACR knowledge domain. Regarding clinical care paths, ChatGPT-4 performs better in diagnosis, prognosis, and toxicity than brachytherapy and dosimetry. It lacks proficiency in in-depth details of clinical trials. For the Gray Zone cases, ChatGPT-4 is able to suggest a personalized treatment approach to each case with high correctness and comprehensiveness. Importantly, it provides novel treatment aspects for many cases, which are not suggested by any human experts. Both evaluations demonstrate the potential of ChatGPT-4 in medical education for the general public and cancer patients, as well as the potential to aid clinical decision-making, while acknowledging its limitations in certain domains. Because of the risk of hallucination, facts provided by ChatGPT always need to be verified.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 68 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

26 of 68 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
14 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  3. 2019Publicly Available Clinical BERT Embeddingscited by 995
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2016Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentationcited by 263
  6. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  7. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  8. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  9. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  10. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  11. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  12. 2023Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Noveltycited by 27
  13. 2021Deep learning for brain metastasis detection and segmentation in longitudinal MRI datacited by 15
  14. 2022Deep learning for automatic head and neck lymph node level delineation provides expert-level accuracycited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Large Language Models in Fire Engineering: An Examination of Technical Questions Against Domain Knowledgecited by 5
  2. 2023Deep Learning for Cancer Prognosis Prediction Using Portrait Photos by StyleGAN Embeddingcited by 2
  3. 2024Tipta uzmanlik sinavinda (tus) buyuk dil modelleri insanlardan daha mi basarili?cited by 0