mlsanityData to 7 Oct 2026

Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

arXiv 2609.00296 ↗First posted 31 Aug 2026cs.CLcited by 0

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Large language model (LLM) agents are increasingly proposed for healthcare tasks such as clinical documentation, evidence retrieval, patient messaging, and care coordination. Yet many evaluations remain limited to static medical question answering or one-shot generation, under-representing longitudinal state, interruptions, and human handoffs. We introduce an episode-level evaluation protocol for healthcare NLP agents. The protocol separates evidence across model, agent, and simulated-workflow behavior; specifies a five-field episode schema; and defines annotation and scoring for state continuity, evidence traceability, and escalation decisions. It is instantiated as four task templates: documentation update, evidence retrieval, patient messaging, and triage handoff. The protocol does not claim to measure clinical outcomes or deployment value. Instead, it supplies a reproducible intermediate evaluation layer between static benchmarks and prospective workflow studies, with an explicit cost-sensitive treatment of missed versus unnecessary escalation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 36 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

31 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments4 May
  2. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents24 Jan

Other references · 29

  1. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  2. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  3. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2025MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilotcited by 83
  6. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
  7. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  8. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 58
  9. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  10. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  11. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  12. 2025MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Modelscited by 33
17 more
  1. 2023MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Modelscited by 29
  2. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  3. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  4. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 22
  5. 2024MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Modelscited by 20
  6. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Modelscited by 12
  7. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 11
  8. 2025AutoPCR: Automated Phenotype Concept Recognition by Promptingcited by 10
  9. 2026The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agentscited by 10
  10. 2024Evaluating large language models in medical applications: a surveycited by 8
  11. 2025Large Language Models in Healthcarecited by 6
  12. 2026GRASP: Plan-Guided Graph Retrieval with Adaptive Fusion and Reranking on Semi-Structured Knowledge Basescited by 6
  13. 2025Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerationscited by 5
  14. 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routingcited by 4
  15. 2025Retrieval Augmented Generation Evaluation for Health Documentscited by 3
  16. 2026EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMscited by 2
  17. 2026Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Casescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks
  2. 2026ART: Action-based Reasoning Task Benchmarking for Medical AI Agents
  3. 2025AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
  4. 2026PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
  5. 2026When OpenClaw Meets Hospital: Toward an Agentic Operating System for Dynamic Clinical Workflows
  6. 2026Agentic AI in Healthcare & Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-based Agents
  7. 2025CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environment
  8. 2024Advancing Healthcare Automation: Multi-Agent System for Medical Necessity Justification