# Benchmarking emergency department triage prediction models with machine learning and large public electronic health records

arXiv: https://arxiv.org/abs/2111.11017 · HTML: https://mlsanity.com/paper/2111.11017/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 81 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 7 of 7)

- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies](https://mlsanity.com/paper/2107.09951/) (2021)
- [Early hospital mortality prediction using vital signals](https://mlsanity.com/paper/1803.06589/) (2018)
- [AutoScore-Survival: Developing interpretable machine learning-based time-to-event scores with right-censored survival data](https://mlsanity.com/paper/2106.06957/) (2021)
- [AutoScore-Imbalance: An interpretable machine learning tool for development of clinical scores with rare events data](https://mlsanity.com/paper/2107.06039/) (2021)

## Built on by (most cited first, 30 of 31)

- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [MissDiff: Training Diffusion Models on Tabular Data with Missing Values](https://mlsanity.com/paper/2307.00467/) (2023)
- [A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?](https://mlsanity.com/paper/2409.15277/) (2024)
- [Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine](https://mlsanity.com/paper/2311.04937/) (2023)
- [Emergency Department Decision Support using Clinical Pseudo-notes](https://mlsanity.com/paper/2402.00160/) (2024)
- [ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room](https://mlsanity.com/paper/2505.22919/) (2025)
- [Foundation Model of Electronic Medical Records for Adaptive Risk Estimation](https://mlsanity.com/paper/2502.06124/) (2025)
- [Coarse race data conceals disparities in clinical risk score performance](https://mlsanity.com/paper/2304.09270/) (2023)
- [ED-Copilot: Reduce Emergency Department Wait Time with Language Model Diagnostic Assistance](https://mlsanity.com/paper/2402.13448/) (2024)
- [Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches](https://mlsanity.com/paper/2311.03417/) (2023)
- [Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record](https://mlsanity.com/paper/2303.14920/) (2023)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Developing Federated Time-to-Event Scores Using Heterogeneous Real-World Survival Data](https://mlsanity.com/paper/2403.05229/) (2024)
- [Survival modeling using deep learning, machine learning and statistical methods: A comparative analysis for predicting mortality after hospital admission](https://mlsanity.com/paper/2403.06999/) (2024)
- [To which reference class do you belong? Measuring racial fairness of reference classes with normative modeling](https://mlsanity.com/paper/2407.19114/) (2024)
- [SIM-Shapley: A Stable and Computationally Efficient Approach to Shapley Value Approximation](https://mlsanity.com/paper/2505.08198/) (2025)
- [Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks](https://mlsanity.com/paper/2507.02819/) (2025)
- [Towards symbolic regression for interpretable clinical decision scores](https://mlsanity.com/paper/2512.07961/) (2025)
- [Forecasting mortality associated emergency department crowding](https://mlsanity.com/paper/2410.08247/) (2024)
- [Validating Emergency Department Admission Predictions Based on Local Data Through MIMIC-IV](https://mlsanity.com/paper/2503.22706/) (2025)
- [Traversal Learning: A Lossless And Efficient Distributed Learning Framework](https://mlsanity.com/paper/2504.07471/) (2025)
- [LLM-Assisted Emergency Triage Benchmark: Bridging Hospital-Rich and MCI-Like Field Simulation](https://mlsanity.com/paper/2509.26351/) (2025)
- [Stable Prediction of Adverse Events in Medical Time-Series Data](https://mlsanity.com/paper/2510.14286/) (2025)
- [Closing Gaps: An Imputation Analysis of ICU Vital Signs](https://mlsanity.com/paper/2510.24217/) (2025)
- [Communication-efficient distributed hazard difference estimation for heterogeneous multi-site survival data](https://mlsanity.com/paper/2601.14609/) (2026)
- [Benchmarking Early Deterioration Prediction Across Hospital-Rich and MCI-Like Emergency Triage Under Constrained Sensing](https://mlsanity.com/paper/2602.20168/) (2026)
- [ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations](https://mlsanity.com/paper/2604.07019/) (2026)
- [Generating synthetic electronic health record data using agent-based models to evaluate machine learning robustness under mass casualty incidents](https://mlsanity.com/paper/2605.09951/) (2026)
- [ELICITED: EHR-grounded Longitudinal Interactive Conversations for Information-seeking Triage Evaluation and Decision-making](https://mlsanity.com/paper/2608.09024/) (2026)
- [Memorisation bias in medical AI](https://mlsanity.com/paper/2609.17223/) (2026)

## Abstract (arXiv)

The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop predictive models and decision support systems to address these challenges. To date, however, there are no widely accepted benchmark ED triage prediction models based on large-scale public EHR data. An open-source benchmarking platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. In this paper, based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we developed a publicly available benchmark suite for ED triage predictive models and created a benchmark dataset that contains over 400,000 ED visits from 2011 to 2019. We introduced three ED-based outcomes (hospitalization, critical outcomes, and 72-hour ED reattendance) and implemented a variety of popular methodologies, ranging from machine learning methods to clinical scoring systems. We evaluated and compared the performance of these methods against benchmark tasks. Our codes are open-source, allowing anyone with MIMIC-IV-ED data access to perform the same steps in data processing, benchmark model building, and experiments. This study provides future researchers with insights, suggestions, and protocols for managing raw data and developing risk triaging tools for emergency care.