# MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows

arXiv: https://arxiv.org/abs/2603.24649 · HTML: https://mlsanity.com/paper/2603.24649/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 70 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Builds on (most cited first, 30 of 35)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale](https://mlsanity.com/paper/2406.19280/) (2024)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning](https://mlsanity.com/paper/2502.19634/) (2025)
- [MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making](https://mlsanity.com/paper/2404.15155/) (2024)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images](https://mlsanity.com/paper/2111.04740/) (2021)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2408.04187/) (2024)
- [MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow](https://mlsanity.com/paper/2503.18968/) (2025)
- [VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging](https://mlsanity.com/paper/2406.05285/) (2024)
- [An Agentic System for Rare Disease Diagnosis with Traceable Reasoning](https://mlsanity.com/paper/2506.20430/) (2025)
- [The University of California San Francisco Preoperative Diffuse Glioma MRI (UCSF-PDGM) Dataset](https://mlsanity.com/paper/2109.00356/) (2021)
- [MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images](https://mlsanity.com/paper/2203.12362/) (2022)
- [MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports](https://mlsanity.com/paper/2505.11733/) (2025)
- [Capabilities of GPT-5 on Multimodal Medical Reasoning](https://mlsanity.com/paper/2508.08224/) (2025)
- [CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2410.00379/) (2024)
- [CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering](https://mlsanity.com/paper/2505.16229/) (2025)
- [Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning](https://mlsanity.com/paper/2502.07143/) (2025)
- [ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reports](https://mlsanity.com/paper/2507.22030/) (2025)
- [NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI](https://mlsanity.com/paper/2505.14064/) (2025)
- [CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting](https://mlsanity.com/paper/2407.08811/) (2024)

## Built on by (most cited first, 5 of 5)

- [Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve](https://mlsanity.com/paper/2604.14475/) (2026)
- [Healthcare AI GYM for Medical Agents](https://mlsanity.com/paper/2605.02943/) (2026)
- [GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI](https://mlsanity.com/paper/2605.00876/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)
- [Source-Dependent Deference in Medical Imaging Agents Under Falsified Findings: A Pilot Audit](https://mlsanity.com/paper/2608.29800/) (2026)

## Abstract (arXiv)

Medical imaging benchmarks often evaluate VLMs on pre-selected 2D images, slices, crops, or patches, making evaluation closer to visual recognition. Real clinical workflows impose a different burden: readers must search through complete studies, operate imaging software, navigate across slices and magnifications, and document visual evidence that can be audited. We argue that this evidence-producing workflow is a critical missing evaluation axis for medical imaging agents. To study it, we introduce MedFlowBench, a full-study benchmark for VLM agents, together with MedOpenClaw, a controlled and replayable runtime in which agents operate medical imaging viewers such as 3D Slicer and QuPath. In each episode, an agent inspects a complete radiology study or whole-slide pathology image, returns a task answer, and submits structured evidence, including key slices, coordinates, regions of interest, or lesion-state fields. This evidence is automatically checked against withheld masks, annotations, and labels. Across evaluated models, final answer-only scoring gives an overly optimistic picture: when answers must also be supported by correct evidence, performance drops substantially on complex workflows. We further find that adding image-analysis tools does not by itself solve the problem. Tools help when they make a complex procedure simple and reliable, but agents still struggle when they must choose inputs, manage viewer state, and verify intermediate outputs over multiple steps. MedFlowBench exposes whether medical imaging agents can produce auditable evidence from complete studies, rather than plausible answers from selected images.