# ART: Action-based Reasoning Task Benchmarking for Medical AI Agents

arXiv: https://arxiv.org/abs/2601.08988 · HTML: https://mlsanity.com/paper/2601.08988/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 1 of its 9 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 1 of 1)

- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)

## Built on by (most cited first, 3 of 3)

- [Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction](https://mlsanity.com/paper/2605.06191/) (2026)
- [Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare](https://mlsanity.com/paper/2605.08445/) (2026)
- [GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space](https://mlsanity.com/paper/2608.30188/) (2026)

## Abstract (arXiv)

Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over structured electronic health records (EHRs). While large language models (LLMs) show promise in healthcare, existing benchmarks inadequately assess performance on action-based tasks involving threshold evaluation, temporal aggregation, and conditional logic. We introduce ART, an Action-based Reasoning clinical Task benchmark for medical AI agents, which mines real-world EHR data to create challenging tasks targeting known reasoning weaknesses. Through analysis of existing benchmarks, we identify three dominant error categories: retrieval failures, aggregation errors, and conditional logic misjudgments. Our four-stage pipeline -- scenario identification, task generation, quality audit, and evaluation -- produces diverse, clinically validated tasks grounded in real patient data. Evaluating GPT-4o-mini and Claude 3.5 Sonnet on 600 tasks shows near-perfect retrieval after prompt refinement, but substantial gaps in aggregation (28--64%) and threshold reasoning (32--38%). By exposing failure modes in action-oriented EHR reasoning, ART advances toward more reliable clinical agents, an essential step for AI systems that reduce cognitive load and administrative burden, supporting workforce capacity in high-demand care settings