mlsanityData to 14 Oct 2026

Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark

arXiv 2609.30027 ↗First posted 24 Sep 2026cs.AI cs.DBcited by 0

Not on the map. Why?

Abstract (arXiv)

Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real electronic health record (EHR) data cannot be openly shared due to privacy, ethics or data use issues and it does not contain verifiable ground truth since the chart records only reflect what clinicians documented. We introduce Synthetic Hospital, an open, fully synthetic, fact-grounded longitudinal EHR benchmark that resolves the open sharing and verifiable ground truth barriers. Built entirely from public medical-education material with no protected health information, it comprises 1,268 longitudinal patients and 5,602 encounters, where every diagnosis, finding, and temporal relation is grounded in standard ontologies (ICD-10-CM, SNOMED CT, LOINC) and with a complete provenance chain back to its source medical education material. Synthetic Hospital is served through a simulated hospital record system that mirrors real EHR infrastructure (standard interoperability APIs, role-based access and function-calling interface). In a blinded review, physicians distinguished its records from real patient charts at near-chance rates (53%). Across 10 frontier and open models, none approaches ceiling: the best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, level with the mean of seven physicians on a matched subset but well below the best of them (0.89), and misses roughly half of clinically relevant findings when summarizing a chart. Overall, these results highlight that Synthetic Hospital is a difficult and realistic test of clinical AI performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

50 of 67 references.

References · 50

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  7. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  8. 2018Scalable and accurate deep learning for electronic health recordscited by 280
  9. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  10. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 237
  11. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  12. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
38 more
  1. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  2. 2017Generating Multi-label Discrete Patient Records using Generative Adversarial Networkscited by 197
  3. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  4. 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 105
  5. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  6. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 101
  7. 2024Towards Conversational Diagnostic AIcited by 79
  8. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  9. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 62
  10. 2025Sequential Diagnosis with Language Modelscited by 60
  11. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 56
  12. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  13. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  14. 2024People cannot distinguish GPT-4 from a human in a Turing testcited by 48
  15. 2023EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Recordscited by 45
  16. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 38
  17. 2025Capabilities of GPT-5 on Multimodal Medical Reasoningcited by 36
  18. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 35
  19. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 34
  20. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Casescited by 29
  21. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  22. 2024CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelinescited by 22
  23. 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Roomcited by 22
  24. 2025FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answeringcited by 22
  25. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environmentscited by 21
  26. 2019Towards Automatic Generation of Shareable Synthetic Clinical Notes Using Neural Language Modelscited by 20
  27. 2024Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicinecited by 13
  28. 2026HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Taskscited by 13
  29. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 12
  30. 2023Overview of the Problem List Summarization (ProbSum) 2023 Shared Task on Summarizing Patients' Active Diagnoses and Problems from Electronic Health Record Progress Notescited by 11
  31. 2023Synthesize High-dimensional Longitudinal Electronic Health Records via Hierarchical Autoregressive Language Modelcited by 8
  32. 2025It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Educationcited by 7
  33. 2024SimSUM: Simulated Benchmark with Structured and Unstructured Medical Recordscited by 6
  34. 2026EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoningcited by 5
  35. 2026ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agentscited by 4
  36. 2026MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agentscited by 4
  37. 2025CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessmentcited by 3
  38. 2026From Statistical Fidelity to Clinical Consistency: Scalable Generation and Auditing of Synthetic Patient Trajectoriescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
  2. 2026EHR-RobustGym: Benchmarking and Training Agents for Robust Clinical Reasoning
  3. 2025EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks
  4. 2026EHRWorld: A Patient-Centric Medical World Model for Long-Horizon Clinical Trajectories
  5. 2025A Generative Approach for Semantic Auditing of Electronic Health Records
  6. 2025Patient-Zero: Scaling Synthetic Patient Agents to Real-World Distributions without Real Patient Data
  7. 2026A Multi-Agent Pipeline for Source-Grounded Synthetic Note Generation from Longitudinal Structured EHR
  8. 2026EHR2Trace: Auditable EHR Data Infrastructure for Patient World Models and Clinical Agents