mlsanityData to 7 Oct 2026

Evidence Is All You Need: Ordering Imaging Studies via Language Model Alignment with the ACR Appropriateness Criteria

arXiv 2409.19177 ↗First posted 27 Sep 2024cs.LG cs.CL cs.CYcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment

Abstract (arXiv)

Diagnostic imaging studies are an increasingly important component of the workup and management of acutely presenting patients. However, ordering appropriate imaging studies according to evidence-based medical guidelines is a challenging task with a high degree of variability between healthcare providers. To address this issue, recent work has investigated if generative AI and large language models can be leveraged to help clinicians order relevant imaging studies for patients. However, it is challenging to ensure that these tools are correctly aligned with medical guidelines, such as the American College of Radiology's Appropriateness Criteria (ACR AC). In this study, we introduce a framework to intelligently leverage language models by recommending imaging studies for patient cases that are aligned with evidence-based guidelines. We make available a novel dataset of patient "one-liner" scenarios to power our experiments, and optimize state-of-the-art language models to achieve an accuracy on par with clinicians in image ordering. Finally, we demonstrate that our language model-based pipeline can be used as intelligent assistants by clinicians to support image ordering workflows and improve the accuracy of imaging study ordering according to the ACR AC. Our work demonstrates and validates a strategy to leverage AI-based software to improve trustworthy clinical decision making in alignment with expert evidence-based guidelines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 110 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

17 of 110 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2024Mixtral of Expertscited by 1,785
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  12. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
5 more
  1. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  2. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 104
  3. 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicinecited by 83
  4. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 58
  5. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
  2. 2025Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model
  3. 2026A radiographic world model for clinical reasoning and evidence generation
  4. 2026MedScribe: Clinically Grounded CT Reporting through Agentic Workflows
  5. 2026Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs
  6. 2026Medical Context Distorts Decisions in Clinical Vision Language Models
  7. 2025RadHiera: Semantic Hierarchical Reinforcement Learning for Medical Report Generation
  8. 2023Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation