# SimSUM: Simulated Benchmark with Structured and Unstructured Medical Records

arXiv: https://arxiv.org/abs/2409.08936 · HTML: https://mlsanity.com/paper/2409.08936/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 58 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health](https://mlsanity.com/paper/2306.10070/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Causal Machine Learning for Healthcare and Precision Medicine](https://mlsanity.com/paper/2205.11402/) (2022)
- [Multimodal Machine Learning for Automated ICD Coding](https://mlsanity.com/paper/1810.13348/) (2018)
- [Natural Language Generation for Electronic Health Records](https://mlsanity.com/paper/1806.01353/) (2018)
- [Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges](https://mlsanity.com/paper/2407.00116/) (2024)
- [A Comprehensive Scoping Review of Bayesian Networks in Healthcare: Past, Present and Future](https://mlsanity.com/paper/2002.08627/) (2020)
- [PromptEHR: Conditional Electronic Healthcare Records Generation with Prompt Learning](https://mlsanity.com/paper/2211.01761/) (2022)
- [The Three Ghosts of Medical AI: Can the Black-Box Present Deliver?](https://mlsanity.com/paper/2012.06000/) (2020)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)
- [BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance](https://mlsanity.com/paper/2305.13395/) (2023)
- [Clinical Reasoning over Tabular Data and Text with Bayesian Networks](https://mlsanity.com/paper/2403.09481/) (2024)
- [EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records](https://mlsanity.com/paper/2406.16341/) (2024)

## Built on by (most cited first, 6 of 6)

- [Dynamic Expert-Guided Model Averaging for Causal Discovery](https://mlsanity.com/paper/2601.16715/) (2026)
- [From Text to Treatment Effects: A Meta-Learning Approach to Handling Text-Based Confounding](https://mlsanity.com/paper/2409.15503/) (2024)
- [Patient-level Information Extraction by Consistent Integration of Textual and Tabular Evidence with Bayesian Networks](https://mlsanity.com/paper/2511.17056/) (2025)
- [Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis](https://mlsanity.com/paper/2606.19183/) (2026)
- [Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark](https://mlsanity.com/paper/2609.30027/) (2026)
- [Diagnosing and Improving Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2609.38005/) (2026)

## Abstract (arXiv)

Clinical information extraction, which involves structuring clinical concepts from unstructured medical text, remains a challenging problem that could benefit from the inclusion of tabular background information available in electronic health records. Existing open-source datasets lack explicit links between structured features and clinical concepts in the text, motivating the need for a new research dataset. We introduce SimSUM, a benchmark dataset of 10,000 simulated patient records that link unstructured clinical notes with structured background variables. Each record simulates a patient encounter in the domain of respiratory diseases and includes tabular data (e.g., symptoms, diagnoses, underlying conditions) generated from a Bayesian network whose structure and parameters are defined by domain experts. A large language model (GPT-4o) is prompted to generate a clinical note describing the encounter, including symptoms and relevant context. These notes are annotated with span-level symptom mentions. We conduct an expert evaluation to assess note quality and run baseline predictive models on both the tabular and textual data. The SimSUM dataset is primarily designed to support research on clinical information extraction in the presence of tabular background variables, which can be linked through domain knowledge to concepts of interest to be extracted from the text -- namely, symptoms in the case of SimSUM. Secondary uses include research on the automation of clinical reasoning over both tabular data and text, causal effect estimation in the presence of tabular and/or textual confounders, and multi-modal synthetic data generation. SimSUM is not intended for training clinical decision support systems or production-grade models, but rather to facilitate reproducible research in a simplified and controlled setting.