# Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance

arXiv: https://arxiv.org/abs/2606.18613 · HTML: https://mlsanity.com/paper/2606.18613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [DDXPlus: A New Dataset For Automatic Medical Diagnosis](https://mlsanity.com/paper/2205.09148/) (2022)
- [The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence](https://mlsanity.com/paper/2605.26494/) (2026)
- [NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes](https://mlsanity.com/paper/2310.15959/) (2023)
- [ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?](https://mlsanity.com/paper/2411.06469/) (2024)
- [A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains](https://mlsanity.com/paper/2507.23486/) (2025)
- [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025)
- [FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering](https://mlsanity.com/paper/2509.19319/) (2025)
- [LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models](https://mlsanity.com/paper/2403.14066/) (2024)
- [Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM](https://mlsanity.com/paper/2408.06285/) (2024)
- [DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversation](https://mlsanity.com/paper/2408.06044/) (2024)
- [S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework](https://mlsanity.com/paper/2508.02082/) (2025)
- [A Mention-Ranking Model for Abstract Anaphora Resolution](https://mlsanity.com/paper/1706.02256/) (2017)
- [RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Game](https://mlsanity.com/paper/2604.10740/) (2026)

## Abstract (arXiv)

The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated capabilities: clinical knowledge, EHR system interaction, or patient communication. Physician assistance instead requires coordinating these capabilities within the same interaction, where physicians issue underspecified requests, patients describe symptoms ambiguously, and EHR systems demand precise tool use. We introduce PhysAssistBench, a benchmark for interactive doctor-patient-EHR assistance. Built from real MIMIC-IV cases, PhysAssistBench uses a scalable pipeline to construct agentic patients: interactive, record-grounded agents that turn static EHR records into multi-turn clinical scenarios while preserving clinical factuality. PhysAssistBench provides a curated bilingual evaluation set of 1,296 manually reviewed and physician-validated turns. Experiments with leading LLMs show that current models remain unreliable in this setting, which exposes a key bottleneck for clinical LLMs: reliable assistance requires coordination across knowledge, communication, and systems, not isolated gains in any of them.