# The Consequences of the Framing of Machine Learning Risk Prediction Models: Evaluation of Sepsis in General Wards

arXiv: https://arxiv.org/abs/2101.10790 · HTML: https://mlsanity.com/paper/2101.10790/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 51 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier](https://mlsanity.com/paper/1706.04152/) (2017)
- [Explainable artificial intelligence model to predict acute critical illness from electronic health records](https://mlsanity.com/paper/1912.01266/) (2019)
- [DeepSOFA: A Continuous Acuity Score for Critically Ill Patients using Clinically Interpretable Deep Learning](https://mlsanity.com/paper/1802.10238/) (2018)
- [An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection](https://mlsanity.com/paper/1708.05894/) (2017)
- [Early detection of sepsis utilizing deep learning on electronic health record event sequences](https://mlsanity.com/paper/1906.02956/) (2019)
- [Leveraging Implicit Expert Knowledge for Non-Circular Machine Learning in Sepsis Prediction](https://mlsanity.com/paper/1909.09557/) (2019)
- [From Bit To Bedside: A Practical Framework For Artificial Intelligence Product Development In Healthcare](https://mlsanity.com/paper/2003.10303/) (2020)
- [Early Recognition of Sepsis with Gaussian Process Temporal Convolutional Networks and Dynamic Time Warping](https://mlsanity.com/paper/1902.01659/) (2019)

## Built on by (most cited first, 1 of 1)

- [NPRL: Nightly Profile Representation Learning for Early Sepsis Onset Prediction in ICU Trauma Patients](https://mlsanity.com/paper/2304.12737/) (2023)

## Abstract (arXiv)

Objectives: To evaluate the consequences of the framing of machine learning risk prediction models. We evaluate how framing affects model performance and model learning in four different approaches previously applied in published artificial-intelligence (AI) models. Setting and participants: We analysed structured secondary healthcare data from 221,283 citizens from four Danish municipalities who were 18 years of age or older. Results: The four models had similar population level performance (a mean area under the receiver operating characteristic curve of 0.73 to 0.82), in contrast to the mean average precision, which varied greatly from 0.007 to 0.385. Correspondingly, the percentage of missing values also varied between framing approaches. The on-clinical-demand framing, which involved samples for each time the clinicians made an early warning score assessment, showed the lowest percentage of missing values among the vital sign parameters, and this model was also able to learn more temporal dependencies than the others. The Shapley additive explanations demonstrated opposing interpretations of SpO2 in the prediction of sepsis as a consequence of differentially framed models. Conclusions: The profound consequences of framing mandate attention from clinicians and AI developers, as the understanding and reporting of framing are pivotal to the successful development and clinical implementation of future AI technology. Model framing must reflect the expected clinical environment. The importance of proper problem framing is by no means exclusive to sepsis prediction and applies to most clinical risk prediction models.