Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
Clinical image interpretation is inherently multi-step and tool-centric: clinicians iteratively combine visual evidence with patient context, quantify findings, and refine their decisions through a sequence of specialized procedures. While LLM-based agents promise to orchestrate such heterogeneous medical tools, existing systems treat tool sets and invocation strategies as static after deployment. This design is brittle under real-world domain shifts, across tasks, and evolving diagnostic requirements, where predefined tool chains frequently degrade and demand costly manual re-design. We propose MACRO, a self-evolving, experience-augmented medical agent that shifts from static tool composition to experience-driven tool discovery. From verified execution trajectories, the agent autonomously identifies recurring effective multi-step tool sequences, synthesizes them into reusable composite tools, and registers these as new high-level primitives that continuously expand its behavioral repertoire. A lightweight image-feature memory grounds tool selection in a visual-clinical context, while a GRPO-like training loop reinforces reliable invocation of discovered composites, enabling closed-loop self-improvement with minimal supervision. Extensive experiments across diverse medical imaging datasets and tasks demonstrate that autonomous composite tool discovery consistently improves multi-step orchestration accuracy and cross-domain generalization over strong baselines and recent state-of-the-art agentic methods, bridging the gap between brittle static tool use and adaptive, context-aware clinical AI assistance. Code will be available upon acceptance.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 54 references that are in the map. First posted on arXiv 6 Mar 2026.
Builds on
39 of 54 references.
Closest ancestors · 5
Same direction, newest first
- 2025Medical AI Consensus: A Multi-Agent Framework for Radiology Report Generation and Evaluation22 Sep
- 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions28 Mar
- 2024Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence2 Dec
- 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agent2 Jul
- 2024Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis29 Jan
Other references · 34
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2023GPT-4 Technical Reportcited by 18,132
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2023A Survey of Large Language Modelscited by 2,537
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
22 more
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
- 2023Towards Generalist Biomedical AIcited by 286
- 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
- 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
- 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
- 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
- 2024MedMamba: Vision Mamba for Medical Image Classificationcited by 86
- 2024Convolutional Kolmogorov-Arnold Networkscited by 76
- 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
- 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
- 2024CoD, Towards an Interpretable Medical Agent using Chain of Diagnosiscited by 43
- 2022REFUGE2 Challenge: A Treasure Trove for Multi-Dimension Analysis and Evaluation in Glaucoma Screeningcited by 37
- 2023Large Language Models Streamline Automated Machine Learning for Clinical Studiescited by 24
- 2025Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoningcited by 16
- 2024Almanac Copilot: Towards Autonomous Electronic Health Record Navigationcited by 9
- 2025RAM-W600: A Multi-Task Wrist Dataset and Benchmark for Rheumatoid Arthritiscited by 5
- 2025Retrieval-Augmented Framework for LLM-Based Clinical Decision Supportcited by 2
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Librariescited by 6
- 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generationcited by 5
- 2026The Last AI Built by Humans: Toward Genuine Recursive Self-Improvementcited by 5
- 2026NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Researchcited by 3
- 2026Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answeringcited by 2
- 2026Workflow Closure Is Not Scientific Closure in Auto-Research Systemscited by 1
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
- 2026MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolutioncited by 0
- 2026ReVision3D: Attribution-Guided Recursive Self-Improvement for 3D Medical Perceptioncited by 0
- 2026From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoningcited by 0
- 2026From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning30 Sep 2026
- 2026ReVision3D: Attribution-Guided Recursive Self-Improvement for 3D Medical Perception26 Sep 2026
- 2026MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution21 Sep 2026
- 2026The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement10 Sep 2026
- 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy13 Jul 2026
- 2026Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries11 Jul 2026
- 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation6 Jun 2026
- 2026Workflow Closure Is Not Scientific Closure in Auto-Research Systems25 May 2026
- 2026NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Research7 May 2026
- 2026Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering14 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026Towards Autonomous and Auditable Medical Imaging Model Development
- 2026Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve
- 2025Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization
- 2026Can Coding Agents Build Robust Baselines? A Skill-Based Approach for Automating the Medical Imaging Model-Development Pipeline
- 2025ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges
- 2025M^3Builder: A Multi-Agent System for Automated Machine Learning in Medical Imaging
- 2026Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imaging
- 2026MIRA: Medical Image Reflection for Agentic Diagnosis