# AURA: A Multi-Modal Medical Agent for Understanding, Reasoning & Annotation

arXiv: https://arxiv.org/abs/2507.16940 · HTML: https://mlsanity.com/paper/2507.16940/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 22 Jul 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 13 of 13)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation](https://mlsanity.com/paper/2401.12208/) (2024)
- [MedRAX: Medical Reasoning Agent for Chest X-ray](https://mlsanity.com/paper/2502.02673/) (2025)
- [DenseNet: Implementing Efficient ConvNet Descriptor Pyramids](https://mlsanity.com/paper/1404.1869/) (2014)
- [CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets](https://mlsanity.com/paper/2009.05199/) (2020)
- [DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations](https://mlsanity.com/paper/2405.09288/) (2024)

## Built on by (most cited first, 5 of 5)

- [MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2601.07107/) (2026)
- [NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Research](https://mlsanity.com/paper/2605.06584/) (2026)
- [GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI](https://mlsanity.com/paper/2605.00876/) (2026)
- [Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning](https://mlsanity.com/paper/2607.07761/) (2026)
- [CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment](https://mlsanity.com/paper/2610.01166/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have catalyzed a paradigm shift from static prediction systems to agentic AI agents capable of reasoning, interacting with tools, and adapting to complex tasks. While LLM-based agentic systems have shown promise across many domains, their application to medical imaging remains in its infancy. In this work, we introduce AURA, the first visual linguistic explainability agent designed specifically for comprehensive analysis, explanation, and evaluation of medical images. By enabling dynamic interactions, contextual explanations, and hypothesis testing, AURA represents a significant advancement toward more transparent, adaptable, and clinically aligned AI systems. We highlight the promise of agentic AI in transforming medical image analysis from static predictions to interactive decision support. Leveraging Qwen-32B, an LLM-based architecture, AURA integrates a modular toolbox comprising: (i) a segmentation suite with phase grounding, pathology segmentation, and anatomy segmentation to localize clinically meaningful regions; (ii) a counterfactual image-generation module that supports reasoning through image-level explanations; and (iii) a set of evaluation tools including pixel-wise difference-map analysis, classification, and advanced state-of-the-art components to assess diagnostic relevance and visual interpretability.