# Signal Fidelity Index-Aware Calibration for Dementia Predictions Across Heterogeneous Real-World Data

arXiv: https://arxiv.org/abs/2509.08679 · HTML: https://mlsanity.com/paper/2509.08679/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 7 of 7)

- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks](https://mlsanity.com/paper/1908.00690/) (2019)
- [Quantifying Diagnostic Signal Decay in Dementia: A National Study of Medicare Hospitalization Data](https://mlsanity.com/paper/2506.14669/) (2025)

## Abstract (arXiv)

Background: Machine learning models trained on electronic health records (EHRs) often degrade across healthcare systems due to distributional shift. A fundamental but underexplored factor is diagnostic signal decay: variability in diagnostic quality and consistency across institutions, which affects the reliability of codes used for training and prediction. Objective: To develop a Signal Fidelity Index (SFI) quantifying diagnostic data quality at the patient level in dementia, and to test SFI-aware calibration for improving model performance across heterogeneous datasets without outcome labels. Methods: We built a simulation framework generating 2,500 synthetic datasets, each with 1,000 patients and realistic demographics, encounters, and coding patterns based on dementia risk factors. The SFI was derived from six interpretable components: diagnostic specificity, temporal consistency, entropy, contextual concordance, medication alignment, and trajectory stability. SFI-aware calibration applied a multiplicative adjustment, optimized across 50 simulation batches. Results: At the optimal parameter (α = 2.0), SFI-aware calibration significantly improved all metrics (p < 0.001). Gains ranged from 10.3% for Balanced Accuracy to 32.5% for Recall, with notable increases in Precision (31.9%) and F1-score (26.1%). Performance approached reference standards, with F1-score and Recall within 1% and Balanced Accuracy and Detection Rate improved by 52.3% and 41.1%, respectively. Conclusions: Diagnostic signal decay is a tractable barrier to model generalization. SFI-aware calibration provides a practical, label-free strategy to enhance prediction across healthcare contexts, particularly for large-scale administrative datasets lacking outcome labels.