mlsanityData to 14 Oct 2026

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv 2605.30295 ↗First posted 28 May 2026cs.CL cs.AIcited by 1

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in structured, electronic health record-congruent settings remains limited. Existing benchmarks often rely on static datasets or unstructured inputs that do not reflect the interoperable data formats used in clinical systems. We introduce a reusable pipeline for generating terminology-grounded HL7 FHIR R4 bundles from unstructured text, enabling controllable evaluation of clinical decision support systems over structured inputs. The pipeline combines staged LLM generation with terminology-grounded validation and repair to eliminate hallucinated codes and enforce structural and semantic consistency. Applying this approach to MedCaseReasoning, we construct MedCase-Structured, a synthetic dataset of 1,732 FHIR bundles derived from clinician-authored diagnostic cases, producing complete, valid bundles for 97.1% of attempted cases. Evaluation on MedCase-Structured reveals consistently lower diagnostic accuracy for LLMs on structured FHIR inputs than with plain text, highlighting the importance of deployment-aligned benchmarking.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 6 of its 28 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

6 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports16 May

Other references · 5

  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 237
  3. 2025FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answeringcited by 22
  4. 2025EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Taskscited by 6
  5. 2026Evaluation format, not model capability, drives measured triage failure in the assessment of consumer health AIcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency
  2. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  3. 2026CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
  4. 2025Enhancing Clinical Decision Support and EHR Insights through LLMs and the Model Context Protocol: An Open-Source MCP-FHIR Framework
  5. 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
  6. 2025MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs
  7. 2026EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation
  8. 2026ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning