# Avoiding C-hacking when evaluating survival distribution predictions with discrimination measures

arXiv: https://arxiv.org/abs/2112.04828 · HTML: https://mlsanity.com/paper/2112.04828/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 11 of 11)

- [Time-to-Event Prediction with Neural Networks and Cox Regression](https://mlsanity.com/paper/1907.00825/) (2019)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [A Scalable Discrete-Time Survival Model for Neural Networks](https://mlsanity.com/paper/1805.00917/) (2018)
- [Continuous and Discrete-Time Survival Prediction with Neural Networks](https://mlsanity.com/paper/1910.06724/) (2019)
- [Effective Ways to Build and Evaluate Individual Survival Distributions](https://mlsanity.com/paper/1811.11347/) (2018)
- [Countdown Regression: Sharp and Calibrated Survival Predictions](https://mlsanity.com/paper/1806.08324/) (2018)
- [Gaussian Processes for Survival Analysis](https://mlsanity.com/paper/1611.00817/) (2016)
- [Large-scale benchmark study of survival prediction methods using multi-omics data](https://mlsanity.com/paper/2003.03621/) (2020)
- [BDNNSurv: Bayesian deep neural networks for survival analysis using pseudo values](https://mlsanity.com/paper/2101.03170/) (2021)
- [Boosting the concordance index for survival data - a unified framework to derive and evaluate biomarker combinations](https://mlsanity.com/paper/1307.6417/) (2013)
- [Survival analysis as a classification problem](https://mlsanity.com/paper/1909.11171/) (2019)

## Built on by (most cited first, 7 of 7)

- [Deep Learning for Survival Analysis: A Review](https://mlsanity.com/paper/2305.14961/) (2023)
- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [Proper Scoring Rules for Survival Analysis](https://mlsanity.com/paper/2305.00621/) (2023)
- [A Large-Scale Neutral Comparison Study of Survival Models on Low-Dimensional Data](https://mlsanity.com/paper/2406.04098/) (2024)
- [Flexible Group Fairness Metrics for Survival Analysis](https://mlsanity.com/paper/2206.03256/) (2022)
- [The C-index Multiverse](https://mlsanity.com/paper/2508.14821/) (2025)
- [Optimizing Prognostic Biomarker Discovery in Pancreatic Cancer Through Hybrid Ensemble Feature Selection and Multi-Omics Data](https://mlsanity.com/paper/2509.02648/) (2025)

## Abstract (arXiv)

In this paper we consider how to evaluate survival distribution predictions with measures of discrimination. This is a non-trivial problem as discrimination measures are the most commonly used in survival analysis and yet there is no clear method to derive a risk prediction from a distribution prediction. We survey methods proposed in literature and software and consider their respective advantages and disadvantages. Whilst distributions are frequently evaluated by discrimination measures, we find that the method for doing so is rarely described in the literature and often leads to unfair comparisons. We find that the most robust method of reducing a distribution to a risk is to sum over the predicted cumulative hazard. We recommend that machine learning survival analysis software implements clear transformations between distribution and risk predictions in order to allow more transparent and accessible model evaluation. The code used in the final experiment is available at https://github.com/RaphaelS1/distribution_discrimination.