# Evaluating Model Performance in Medical Datasets Over Time

arXiv: https://arxiv.org/abs/2305.13426 · HTML: https://mlsanity.com/paper/2305.13426/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 51 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 13 of 13)

- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [Learning to Diagnose with LSTM Recurrent Neural Networks](https://mlsanity.com/paper/1511.03677/) (2015)
- [Evaluating time series forecasting models: An empirical study on performance estimation methods](https://mlsanity.com/paper/1905.11744/) (2019)
- [Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks](https://mlsanity.com/paper/1908.00690/) (2019)
- [Domain Adaptation under Missingness Shift](https://mlsanity.com/paper/2211.02093/) (2022)
- [Uncertainty-Aware Lookahead Factor Models for Quantitative Investing](https://mlsanity.com/paper/2007.04082/) (2020)
- [Improving Factor-Based Quantitative Investing by Forecasting Company Fundamentals](https://mlsanity.com/paper/1711.04837/) (2017)
- [Treatment Policy Learning in Multiobjective Settings with Fully Observed Outcomes](https://mlsanity.com/paper/2006.00927/) (2020)
- [Learning Clinical Concepts for Predicting Risk of Progression to Severe COVID-19](https://mlsanity.com/paper/2208.13126/) (2022)

## Built on by (most cited first, 4 of 4)

- [What Do Machine Learning Researchers Mean by "Reproducible"?](https://mlsanity.com/paper/2412.03854/) (2024)
- [Time Matters: Examine Temporal Effects on Biomedical Language Models](https://mlsanity.com/paper/2407.17638/) (2024)
- [MoCo-Transfer: Investigating out-of-distribution contrastive learning for limited-data domains](https://mlsanity.com/paper/2311.09401/) (2023)
- [Can We Reliably Rank Model Performance across Domains without Labeled Data?](https://mlsanity.com/paper/2510.09519/) (2025)

## Abstract (arXiv)

Machine learning (ML) models deployed in healthcare systems must face data drawn from continually evolving environments. However, researchers proposing such models typically evaluate them in a time-agnostic manner, splitting datasets according to patients sampled randomly throughout the entire study time period. This work proposes the Evaluation on Medical Datasets Over Time (EMDOT) framework, which evaluates the performance of a model class across time. Inspired by the concept of backtesting, EMDOT simulates possible training procedures that practitioners might have been able to execute at each point in time and evaluates the resulting models on all future time points. Evaluating both linear and more complex models on six distinct medical data sources (tabular and imaging), we show how depending on the dataset, using all historical data may be ideal in many cases, whereas using a window of the most recent data could be advantageous in others. In datasets where models suffer from sudden degradations in performance, we investigate plausible explanations for these shocks. We release the EMDOT package to help facilitate further works in deployment-oriented evaluation over time.