mlsanityData to 14 Oct 2026

A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options

arXiv 2412.10622 ↗First posted 14 Dec 2024physics.med-ph cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Purpose: We present an updated study evaluating the performance of large language models (LLMs) in answering radiation oncology physics questions, focusing on the recently released models. Methods: A set of 100 multiple-choice radiation oncology physics questions, previously created by a well-experienced physicist, was used for this study. The answer options of the questions were randomly shuffled to create "new" exam sets. Five LLMs -- OpenAI o1-preview, GPT-4o, LLaMA 3.1 (405B), Gemini 1.5 Pro, and Claude 3.5 Sonnet -- with the versions released before September 30, 2024, were queried using these new exam sets. To evaluate their deductive reasoning ability, the correct answer options in the questions were replaced with "None of the above." Then, the explain-first and step-by-step instruction prompts were used to test if this strategy improved their reasoning ability. The performance of the LLMs was compared with the answers from medical physicists. Results: All models demonstrated expert-level performance on these questions, with o1-preview even surpassing medical physicists with a majority vote. When replacing the correct answer options with 'None of the above', all models exhibited a considerable decline in performance, suggesting room for improvement. The explain-first and step-by-step instruction prompts helped enhance the reasoning ability of the LLaMA 3.1 (405B), Gemini 1.5 Pro, and Claude 3.5 Sonnet models. Conclusion: These recently released LLMs demonstrated expert-level performance in answering radiation oncology physics questions, exhibiting great potential to assist in radiation oncology physics education and training.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

18 of 42 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2024GPT-4o System Cardcited by 4,588
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
6 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  4. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  5. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  6. 2024The Radiation Oncology NLP Databasecited by 2

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025RadOnc-GPT: An Autonomous LLM Agent for Real-Time Patient Outcomes Labeling at Scalecited by 5
  2. 2025Large Language Models for Bioinformaticscited by 3
  3. 2025Evaluating The Performance of Using Large Language Models to Automate Summarization of CT Simulation Orders in Radiation Oncologycited by 3
  4. 2025Benchmarking GPT-5 in Radiation Oncology: Measurable Gains, but Persistent Need for Expert Oversightcited by 0
  5. 2025An Automated Retrieval-Augmented Generation LLaMA-4 109B-based System for Evaluating Radiotherapy Treatment Planscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning
  2. 2025Multi-Physics: A Comprehensive Benchmark for Multimodal LLMs Reasoning on Chinese Multi-Subject Physics Problems
  3. 2024From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT
  4. 2025Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology
  5. 2024Evaluating ChatGPT on Nuclear Domain-Specific Data
  6. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment
  7. 2026CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine
  8. 2026Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?