# Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML

arXiv: https://arxiv.org/abs/2306.05109 · HTML: https://mlsanity.com/paper/2306.05109/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2023.

## Built on by (most cited first, 29 of 29)

- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [ACES: Automatic Cohort Extraction System for Event-Stream Datasets](https://mlsanity.com/paper/2406.19653/) (2024)
- [FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection](https://mlsanity.com/paper/2408.09227/) (2024)
- [Dynamic Survival Analysis for Early Event Prediction](https://mlsanity.com/paper/2403.12818/) (2024)
- [How Deep is your Guess? A Fresh Perspective on Deep Learning for Medical Time-Series Imputation](https://mlsanity.com/paper/2407.08442/) (2024)
- [FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models](https://mlsanity.com/paper/2408.10276/) (2024)
- [Towards Foundation Models for Critical Care Time Series](https://mlsanity.com/paper/2411.16346/) (2024)
- [Bridging Data Gaps of Rare Conditions in ICU: A Multi-Disease Adaptation Approach for Clinical Prediction](https://mlsanity.com/paper/2507.06432/) (2025)
- [Can we generate portable representations for clinical time series data using LLMs?](https://mlsanity.com/paper/2603.23987/) (2026)
- [Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications](https://mlsanity.com/paper/2403.18316/) (2024)
- [Domain Generalization and Adaptation in Intensive Care with Anchor Regression](https://mlsanity.com/paper/2507.21783/) (2025)
- [PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning](https://mlsanity.com/paper/2601.16414/) (2026)
- [RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation](https://mlsanity.com/paper/2605.13542/) (2026)
- [Privacy-Preserving Dataset Combination](https://mlsanity.com/paper/2502.05765/) (2025)
- [Improving Representation Learning of Complex Critical Care Data with ICU-BERT](https://mlsanity.com/paper/2502.19593/) (2025)
- [Data-Driven Discovery of Feature Groups in Clinical Time Series](https://mlsanity.com/paper/2511.08260/) (2025)
- [Data reuse enables cost-efficient randomized trials of medical AI models](https://mlsanity.com/paper/2511.08986/) (2025)
- [A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking](https://mlsanity.com/paper/2601.06645/) (2026)
- [ICU-TSB: A Benchmark for Temporal Patient Representation Learning for Unsupervised Stratification into Patient Cohorts](https://mlsanity.com/paper/2506.06192/) (2025)
- [Impute With Confidence: A Framework for Uncertainty Aware Multivariate Time Series Imputation](https://mlsanity.com/paper/2507.09353/) (2025)
- [Towards Self-Supervised Foundation Models for Critical Care Time Series](https://mlsanity.com/paper/2509.19885/) (2025)
- [EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databases](https://mlsanity.com/paper/2510.00549/) (2025)
- [Enabling Granular Subgroup Level Model Evaluations by Generating Synthetic Medical Time Series](https://mlsanity.com/paper/2510.19728/) (2025)
- [Closing Gaps: An Imputation Analysis of ICU Vital Signs](https://mlsanity.com/paper/2510.24217/) (2025)
- [VitalBench: A Rigorous Multi-Center Benchmark for Long-Term Vital Sign Prediction in Intraoperative Care](https://mlsanity.com/paper/2511.13757/) (2025)
- [PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units](https://mlsanity.com/paper/2511.22199/) (2025)
- [When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion](https://mlsanity.com/paper/2602.23614/) (2026)
- [Investigating Data Interventions for Subgroup Fairness: An ICU Case Study](https://mlsanity.com/paper/2604.03478/) (2026)

## Abstract (arXiv)

Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. The intensive care unit (ICU) is a natural habitat for ML given the abundance of available data from electronic health records. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance - often more so than model class - indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations. Software Repository: https://github.com/rvandewater/YAIB.