# ACES: Automatic Cohort Extraction System for Event-Stream Datasets

arXiv: https://arxiv.org/abs/2406.19653 · HTML: https://mlsanity.com/paper/2406.19653/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning](https://mlsanity.com/paper/2207.09858/) (2022)
- [Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML](https://mlsanity.com/paper/2306.05109/) (2023)
- [Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks](https://mlsanity.com/paper/2405.01719/) (2024)
- [Unifying Heterogeneous Electronic Health Records Systems via Text-Based Code Embedding](https://mlsanity.com/paper/2108.03625/) (2021)
- [Deep Contextual Clinical Prediction with Reverse Distillation](https://mlsanity.com/paper/2007.05611/) (2020)
- [A Theory of Dynamic Benchmarks](https://mlsanity.com/paper/2210.03165/) (2022)
- [ImageNot: A contrast with ImageNet preserves model rankings](https://mlsanity.com/paper/2404.02112/) (2024)
- [TemporAI: Facilitating Machine Learning Innovation in Time Domain Tasks for Medicine](https://mlsanity.com/paper/2301.12260/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Collection of Innovations in Medical AI for patient records in 2024](https://mlsanity.com/paper/2503.05768/) (2025)
- [FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records](https://mlsanity.com/paper/2505.16941/) (2025)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [Rethinking Tokenization for Clinical Time Series: When Less is More](https://mlsanity.com/paper/2512.05217/) (2025)
- [PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning](https://mlsanity.com/paper/2601.16414/) (2026)
- [EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databases](https://mlsanity.com/paper/2510.00549/) (2025)
- [EveryQuery: A Promptable Foundation Model for Clinical Prediction Tasks over Electronic Health Records](https://mlsanity.com/paper/2603.07900/) (2026)
- [Rethinking How We Evaluate Methodological Progress in Health AI](https://mlsanity.com/paper/2609.18134/) (2026)
- [EHR2Trace: Auditable EHR Data Infrastructure for Patient World Models and Clinical Agents](https://mlsanity.com/paper/2609.38193/) (2026)

## Abstract (arXiv)

Reproducibility remains a significant challenge in machine learning (ML) for healthcare. Datasets, model pipelines, and even task or cohort definitions are often private in this field, leading to a significant barrier in sharing, iterating, and understanding ML results on electronic health record (EHR) datasets. We address a significant part of this problem by introducing the Automatic Cohort Extraction System (ACES) for event-stream data. This library is designed to simultaneously simplify the development of tasks and cohorts for ML in healthcare and also enable their reproduction, both at an exact level for single datasets and at a conceptual level across datasets. To accomplish this, ACES provides: (1) a highly intuitive and expressive domain-specific configuration language for defining both dataset-specific concepts and dataset-agnostic inclusion or exclusion criteria, and (2) a pipeline to automatically extract patient records that meet these defined criteria from real-world data. ACES can be automatically applied to any dataset in either the Medical Event Data Standard (MEDS) or Event Stream GPT (ESGPT) formats, or to *any* dataset in which the necessary task-specific predicates can be extracted in an event-stream form. ACES has the potential to significantly lower the barrier to entry for defining ML tasks in representation learning, redefine the way researchers interact with EHR datasets, and significantly improve the state of reproducibility for ML studies using this modality. ACES is available at: https://github.com/justin13601/aces.