# How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?

arXiv: https://arxiv.org/abs/2412.09529 · HTML: https://mlsanity.com/paper/2412.09529/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 85 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Radiology-Llama2: Best-in-Class Large Language Model for Radiology](https://mlsanity.com/paper/2309.06419/) (2023)
- [GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaboration](https://mlsanity.com/paper/2404.15127/) (2024)
- [AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning](https://mlsanity.com/paper/2402.13225/) (2024)
- [Large-scale Long-tailed Disease Diagnosis on Radiology Images](https://mlsanity.com/paper/2312.16151/) (2023)
- [Synthetic4Health: Generating Annotated Synthetic Clinical Letters](https://mlsanity.com/paper/2409.09501/) (2024)

## Built on by (most cited first, 10 of 10)

- [An Agentic System for Rare Disease Diagnosis with Traceable Reasoning](https://mlsanity.com/paper/2506.20430/) (2025)
- [End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoning](https://mlsanity.com/paper/2508.15746/) (2025)
- [MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology](https://mlsanity.com/paper/2511.20490/) (2025)
- [Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges](https://mlsanity.com/paper/2510.09404/) (2025)
- [MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows](https://mlsanity.com/paper/2603.24649/) (2026)
- [ABRA: Agent Benchmark for Radiology Applications](https://mlsanity.com/paper/2605.11224/) (2026)
- [Explainable Cross-Disease Reasoning for Cardiovascular Risk Assessment from Low-Dose Computed Tomography](https://mlsanity.com/paper/2511.06625/) (2025)
- [CT-PrepAgent: Bounded Policy and Controlled Execution for Adaptive CT Data Preparation](https://mlsanity.com/paper/2608.01233/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)
- [MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles](https://mlsanity.com/paper/2610.04800/) (2026)

## Abstract (arXiv)

Radiology, with its heterogeneous modalities, anatomies, and evolving protocols, is a natural yet high-stakes testbed for agentic AI. As LLMs grow more capable and tool ecosystems such as MCP and Agent Skills more complex, their capability boundaries in radiology remain unclear. We introduce RadA-BenchPlat, a two-layer benchmark spanning idealized reasoning and real-world execution: a synthetic layer with 2.2k clinician-verified records, 24.2k QA pairs, and 10 tool categories under diverse availability settings; and a real-environment layer pairing 165 2D/3D cases with executable tools for grounding, diagnosis, and report generation. We find that high completion rates under idealized settings do not carry over to real environments, where the best agents trail their upper bound by approximately 15% and drop to 0.327 on long-chain tasks. Prompting strategies (few-shot exemplars, multi-agent coordination) and AutoTB (on-the-fly synthesis of missing tools) mitigate execution drift and breakdowns in difficult scenarios, lifting execution success to 94.5%, yet remain not fully reliable. Overall, our findings suggest that radiology agents are no longer limited to conceptual demonstrations, but are beginning to show potential for real-world applications. Project repository: https://github.com/MAGIC-AI4Med/RadABench