# Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Data

arXiv: https://arxiv.org/abs/2010.01149 · HTML: https://mlsanity.com/paper/2010.01149/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data](https://mlsanity.com/paper/2111.08536/) (2021)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Neighborhood Contrastive Learning Applied to Online Patient Monitoring](https://mlsanity.com/paper/2106.05142/) (2021)
- [Temporal Label Smoothing for Early Event Prediction](https://mlsanity.com/paper/2208.13764/) (2022)
- [SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records](https://mlsanity.com/paper/2110.04943/) (2021)
- [Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data](https://mlsanity.com/paper/2312.11502/) (2023)
- [Modelling Patient Trajectories Using Multimodal Information](https://mlsanity.com/paper/2209.04224/) (2022)
- [Leveraging MIMIC Datasets for Better Digital Health: A Review on Open Problems, Progress Highlights, and Future Promises](https://mlsanity.com/paper/2506.12808/) (2025)
- [Automated Primary Hyperparathyroidism Screening with Neural Networks](https://mlsanity.com/paper/2105.02386/) (2021)
- [SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models](https://mlsanity.com/paper/2108.13672/) (2021)
- [PhilHumans: Benchmarking Machine Learning for Personal Health](https://mlsanity.com/paper/2405.02770/) (2024)
- [Rethinking How We Evaluate Methodological Progress in Health AI](https://mlsanity.com/paper/2609.18134/) (2026)

## Abstract (arXiv)

The Large Scale Visual Recognition Challenge based on the well-known Imagenet dataset catalyzed an intense flurry of progress in computer vision. Benchmark tasks have propelled other sub-fields of machine learning forward at an equally impressive pace, but in healthcare it has primarily been image processing tasks, such as in dermatology and radiology, that have experienced similar benchmark-driven progress. In the present study, we performed a comprehensive review of benchmarks in medical machine learning for structured data, identifying one based on the Medical Information Mart for Intensive Care (MIMIC-III) that allows the first direct comparison of predictive performance and thus the evaluation of progress on four clinical prediction tasks: mortality, length of stay, phenotyping, and patient decompensation. We find that little meaningful progress has been made over a 3 year period on these tasks, despite significant community engagement. Through our meta-analysis, we find that the performance of deep recurrent models is only superior to logistic regression on certain tasks. We conclude with a synthesis of these results, possible explanations, and a list of desirable qualities for future benchmarks in medical machine learning.