mlsanityData to 14 Oct 2026

Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics

arXiv 2304.01938 ↗First posted 1 Apr 2023physics.med-ph cs.CL physics.ed-phcited by 49

Not on the map. Why?

Abstract (arXiv)

We present the first study to investigate Large Language Models (LLMs) in answering radiation oncology physics questions. Because popular exams like AP Physics, LSAT, and GRE have large test-taker populations and ample test preparation resources in circulation, they may not allow for accurately assessing the true potential of LLMs. This paper proposes evaluating LLMs on a highly-specialized topic, radiation oncology physics, which may be more pertinent to scientific and medical communities in addition to being a valuable benchmark of LLMs. We developed an exam consisting of 100 radiation oncology physics questions based on our expertise at Mayo Clinic. Four LLMs, ChatGPT (GPT-3.5), ChatGPT (GPT-4), Bard (LaMDA), and BLOOMZ, were evaluated against medical physicists and non-experts. ChatGPT (GPT-4) outperformed all other LLMs as well as medical physicists, on average. The performance of ChatGPT (GPT-4) was further improved when prompted to explain first, then answer. ChatGPT (GPT-3.5 and GPT-4) showed a high level of consistency in its answer choices across a number of trials, whether correct or incorrect, a characteristic that was not observed in the human test groups. In evaluating ChatGPTs (GPT-4) deductive reasoning ability using a novel approach (substituting the correct answer with "None of the above choices is the correct answer."), ChatGPT (GPT-4) demonstrated surprising accuracy, suggesting the potential presence of an emergent ability. Finally, although ChatGPT (GPT-4) performed well overall, its intrinsic properties did not allow for further improvement when scoring based on a majority vote across trials. In contrast, a team of medical physicists were able to greatly outperform ChatGPT (GPT-4) using a majority vote. This study suggests a great potential for LLMs to work alongside radiation oncology experts as highly knowledgeable assistants.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 29 of its 76 references that are in the map. First posted on arXiv 1 Apr 2023.

Builds on

29 of 76 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
17 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  6. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  8. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  9. 2022Can language models learn from explanations in context?cited by 249
  10. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  11. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  12. 2023When Brain-inspired AI Meets AGIcited by 54
  13. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  14. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 32
  15. 2023Mask-guided BERT for Few Shot Text Classificationcited by 25
  16. 2022Coarse-to-fine Knowledge Graph Domain Adaptation based on Distantly-supervised Iterative Trainingcited by 20
  17. 2022Clinical Prompt Learning with Frozen Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 13; 2025: 5; 2026: 1.

49 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  4. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  5. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 90
  6. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  7. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  8. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  9. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  10. 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
  11. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  12. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
37 more
  1. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 34
  2. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  3. 2023Differentiate ChatGPT-generated and Human-written Medical Textscited by 26
  4. 2023Segment Anything Model (SAM) for Radiation Oncologycited by 24
  5. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answeringcited by 23
  6. 2023Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applicationscited by 21
  7. 2023Large language models in medicine: the potentials and pitfallscited by 21
  8. 2024Large Language Model Agent as a Mechanical Designercited by 21
  9. 2023RadOnc-GPT: A Large Language Model for Radiation Oncologycited by 18
  10. 2024Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understandingcited by 15
  11. 2023LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversationcited by 14
  12. 2023Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challengescited by 14
  13. 2023Ophtha-LLaMA2: A Large Language Model for Ophthalmologycited by 9
  14. 2024PhenoFlow: A Human-LLM Driven Visual Analytics System for Exploring Large and Complex Stroke Datasetscited by 9
  15. 2024Fine-Tuning a Local LLaMA-3 Large Language Model for Automated Privacy-Preserving Physician Letter Generation in Radiation Oncologycited by 8
  16. 2024People over trust AI-generated medical responses and view them to be as valid as doctors, despite low accuracycited by 7
  17. 2025LLM-based Semantic Augmentation for Harmful Content Detectioncited by 7
  18. 2023Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classificationcited by 5
  19. 2023Benchmarking a foundation LLM on its ability to re-label structure names in accordance with the AAPM TG-263 reportcited by 5
  20. 2024Evaluating Search Engines and Large Language Models for Answering Health Questionscited by 5
  21. 2024A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled optionscited by 5
  22. 2025RadOnc-GPT: An Autonomous LLM Agent for Real-Time Patient Outcomes Labeling at Scalecited by 5
  23. 2023Evaluating multiple large language models in pediatric ophthalmologycited by 4
  24. 2023Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncologycited by 3
  25. 2023PharmacyGPT: The AI Pharmacistcited by 3
  26. 2025Large Language Models for Bioinformaticscited by 3
  27. 2025Fine-Tuning Open-Source Large Language Models to Improve Their Performance on Radiation Oncology Tasks: A Feasibility Study to Investigate Their Potential Clinical Applications in Radiation Oncologycited by 3
  28. 2023Professional Certification Benchmark Dataset: The First 500 Jobs For Large Language Modelscited by 2
  29. 2023Leveraging GPT-4 for Food Effect Summarization to Enhance Product-Specific Guidance Development via Iterative Promptingcited by 2
  30. 2024Accuracy and Consistency of LLMs in the Registered Dietitian Exam: The Impact of Prompt Engineering and Knowledge Retrievalcited by 2
  31. 2024Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses from Closed-Domain LLM vs. Clinical Teamscited by 2
  32. 2023A criterion for Artificial General Intelligence: hypothetic-deductive reasoning, tested on ChatGPTcited by 1
  33. 2024From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoTcited by 1
  34. 2023Evaluating AI Vocational Skills Through Professional Testingcited by 0
  35. 2024Is GPT-4 conscious?cited by 0
  36. 2025Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversightcited by 0
  37. 2026The Daily Dose: Workflow-Integrated Large Language Model Automation for Clinical Summarization and Trial Identification in Radiation Oncologycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating ChatGPT on Nuclear Domain-Specific Data
  2. 2025Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology
  3. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment
  4. 2024Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning
  5. 2024Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study
  6. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task
  7. 2026Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
  8. 2024Hybrid Student-Teacher Large Language Model Refinement for Cancer Toxicity Symptom Extraction