# Updating Clinical Risk Stratification Models Using Rank-Based Compatibility: Approaches for Evaluating and Optimizing Clinician-Model Team Performance

arXiv: https://arxiv.org/abs/2308.05619 · HTML: https://mlsanity.com/paper/2308.05619/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 37 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 6 of 6)

- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [An Empirical Analysis of Backward Compatibility in Machine Learning Systems](https://mlsanity.com/paper/2008.04572/) (2020)
- [Modeling and Predicting Trust Dynamics in Human-Robot Teaming: A Bayesian Inference Approach](https://mlsanity.com/paper/2007.13151/) (2020)
- [Mind the Performance Gap: Examining Dataset Shift During Prospective Validation](https://mlsanity.com/paper/2107.13964/) (2021)
- [Human-AI Symbiosis: A Survey of Current Approaches](https://mlsanity.com/paper/2103.09990/) (2021)
- [Personalization in Human-AI Teams: Improving the Compatibility-Accuracy Tradeoff](https://mlsanity.com/paper/2004.02289/) (2020)

## Built on by (most cited first, 1 of 1)

- [Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium](https://mlsanity.com/paper/2403.01628/) (2024)

## Abstract (arXiv)

As data shift or new data become available, updating clinical machine learning models may be necessary to maintain or improve performance over time. However, updating a model can introduce compatibility issues when the behavior of the updated model does not align with user expectations, resulting in poor user-model team performance. Existing compatibility measures depend on model decision thresholds, limiting their applicability in settings where models are used to generate rankings based on estimated risk. To address this limitation, we propose a novel rank-based compatibility measure, CR, and a new loss function that aims to optimize discriminative performance while encouraging good compatibility. Applied to a case study in mortality risk stratification leveraging data from MIMIC, our approach yields more compatible models while maintaining discriminative performance compared to existing model selection techniques, with an increase in CR of 0.019 (95% confidence interval: 0.005, 0.035). This work provides new tools to analyze and update risk stratification models used in clinical care.