# Deep Contextual Clinical Prediction with Reverse Distillation

arXiv: https://arxiv.org/abs/2007.05611 · HTML: https://mlsanity.com/paper/2007.05611/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2020.

## Built on by (most cited first, 16 of 16)

- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records](https://mlsanity.com/paper/2110.04943/) (2021)
- [ACES: Automatic Cohort Extraction System for Event-Stream Datasets](https://mlsanity.com/paper/2406.19653/) (2024)
- [Transformer-based Time-to-Event Prediction for Chronic Kidney Disease Deterioration](https://mlsanity.com/paper/2306.05779/) (2023)
- [MEDS-Tab: Automated tabularization and baseline methods for MEDS datasets](https://mlsanity.com/paper/2411.00200/) (2024)
- [Earthquake Nowcasting with Deep Learning](https://mlsanity.com/paper/2201.01869/) (2021)
- [Large-Scale Study of Temporal Shift in Health Insurance Claims](https://mlsanity.com/paper/2305.05087/) (2023)
- [Mixture of Coupled HMMs for Robust Modeling of Multivariate Healthcare Time Series](https://mlsanity.com/paper/2311.07867/) (2023)
- [Dissecting Medical Referral Mechanisms in Health Services: Role of Physician Professional Networks](https://mlsanity.com/paper/2312.02387/) (2023)
- [ICU Bloodstream Infection Prediction: A Transformer-Based Approach for EHR Analysis](https://mlsanity.com/paper/2405.00819/) (2024)
- [Interpretable Predictive Models for Healthcare via Rational Logistic Regression](https://mlsanity.com/paper/2411.03224/) (2024)
- [SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models](https://mlsanity.com/paper/2108.13672/) (2021)
- [Distillation to Enhance the Portability of Risk Models Across Institutions with Large Patient Claims Database](https://mlsanity.com/paper/2207.02445/) (2022)
- [Closing the Gap in High-Risk Pregnancy Care Using Machine Learning and Human-AI Collaboration](https://mlsanity.com/paper/2305.17261/) (2023)
- [Towards Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects](https://mlsanity.com/paper/2502.04899/) (2025)

## Abstract (arXiv)

Healthcare providers are increasingly using machine learning to predict patient outcomes to make meaningful interventions. However, despite innovations in this area, deep learning models often struggle to match performance of shallow linear models in predicting these outcomes, making it difficult to leverage such techniques in practice. In this work, motivated by the task of clinical prediction from insurance claims, we present a new technique called Reverse Distillation which pretrains deep models by using high-performing linear models for initialization. We make use of the longitudinal structure of insurance claims datasets to develop Self Attention with Reverse Distillation, or SARD, an architecture that utilizes a combination of contextual embedding, temporal embedding and self-attention mechanisms and most critically is trained via reverse distillation. SARD outperforms state-of-the-art methods on multiple clinical prediction outcomes, with ablation studies revealing that reverse distillation is a primary driver of these improvements. Code is available at https://github.com/clinicalml/omop-learn.