mlsanityData to 14 Oct 2026

MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional

arXiv 2605.24699 ↗First posted 23 May 2026cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Most reported gains on agentic-LLM clinical benchmarks are often attributed to prompt engineering, yet our results suggest that larger improvements can come from architectural and engine-level design. We present MDIA, a Multi-agent Diagnostic Intelligence Agent implemented as a 7-node specialty-routed clinical reasoning graph, on the full HealthBench Professional benchmark (n = 525), on a non-fine-tuned LLM. MDIA achieves 0.6272 under OpenAI's GPT-5.4-2026-03-05, which is +3.72 pp above the performance of OpenAI's ChatGPT for Clinicians. The experimental work shows that performance lift is attributable to system architecture: specialty routing, multi-turn context preservation, drug-state safety gating, site-filtered search, length-aware synthesis, and engine-level reliability. These findings support the view that agentic clinical benchmark performance is shaped both by the underlying foundation model and the orchestration architecture. Nevertheless, we also noticed notable differences when using other models as a grader; in particular, when using Gemini 2.5 Pro, MDIA scored 0.6585, which suggests that the choice of grader is a source of variability. Robust evaluation of LLMs would therefore require assessment across several independent grader models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 51 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

40 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments4 May
  2. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents24 Jan
  3. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments13 May

Other references · 37

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
25 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  2. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  3. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  4. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  5. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  7. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  8. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  9. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  10. 2022Teaching language models to support answers with verified quotescited by 292
  11. 2022Can large language models reason about medical questions?cited by 194
  12. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  13. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  14. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 101
  15. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
  16. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  17. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 26
  18. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 23
  19. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  20. 2025LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluationscited by 15
  21. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelinescited by 7
  22. 2026MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?cited by 6
  23. 2025OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queriescited by 5
  24. 2026Medical Reasoning with Large Language Models: A Survey and MR-Benchcited by 4
  25. 2025A Real-World Evaluation of LLM Medication Safety Reviews in NHS Primary Carecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
  2. 2026MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
  3. 2025Learning to Be A Doctor: Searching for Effective Medical Agent Architectures
  4. 2026MedCTA: A Benchmark for Clinical Tool Agents
  5. 2026DeepLens Diagnosis Agent: Agentic Workflow Design Lets a Small Reasoning Model Compete with Frontier LLMs
  6. 2026Agentic AI in medicine: architectures, applications, evaluation, and challenges for clinical translation
  7. 2025MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration
  8. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator