# Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration

arXiv: https://arxiv.org/abs/2102.04110 · HTML: https://mlsanity.com/paper/2102.04110/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 8 Feb 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Generating Multi-label Discrete Patient Records using Generative Adversarial Networks](https://mlsanity.com/paper/1703.06490/) (2017)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [A Question-Entailment Approach to Question Answering](https://mlsanity.com/paper/1901.08079/) (2019)
- [MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare](https://mlsanity.com/paper/1810.09593/) (2018)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [SECTOR: A Neural Model for Coherent Topic Segmentation and Classification](https://mlsanity.com/paper/1902.04793/) (2019)
- [Using Clinical Notes with Time Series Data for ICU Management](https://mlsanity.com/paper/1909.09702/) (2019)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [Towards Automated ICD Coding Using Deep Learning](https://mlsanity.com/paper/1711.04075/) (2017)
- [Learning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICU](https://mlsanity.com/paper/1806.02878/) (2018)
- [Patient representation learning and interpretable evaluation using clinical notes](https://mlsanity.com/paper/1807.01395/) (2018)
- [Deep EHR: Chronic Disease Prediction Using Medical Notes](https://mlsanity.com/paper/1808.04928/) (2018)
- [Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset](https://mlsanity.com/paper/2006.07332/) (2020)
- [Towards unstructured mortality prediction with free-text clinical notes](https://mlsanity.com/paper/1911.08437/) (2019)
- [Self-Attention Enhanced Patient Journey Understanding in Healthcare System](https://mlsanity.com/paper/2006.10516/) (2020)
- [An Analysis of Attention over Clinical Notes for Predictive Tasks](https://mlsanity.com/paper/1904.03244/) (2019)
- [Predicting Multiple ICD-10 Codes from Brazilian-Portuguese Clinical Notes](https://mlsanity.com/paper/2008.01515/) (2020)

## Built on by (most cited first, 30 of 44)

- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2310.19975/) (2023)
- [RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records](https://mlsanity.com/paper/2403.00815/) (2024)
- [Literature-Augmented Clinical Outcome Prediction](https://mlsanity.com/paper/2111.08374/) (2021)
- [Clinical Prompt Learning with Frozen Language Models](https://mlsanity.com/paper/2205.05535/) (2022)
- [Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt](https://mlsanity.com/paper/2211.13813/) (2022)
- [NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs](https://mlsanity.com/paper/2312.11803/) (2023)
- [De-identification is not enough: a comparison between de-identified and synthetic clinical notes](https://mlsanity.com/paper/2402.00179/) (2024)
- [Auditing and Generating Synthetic Data with Controllable Trust Trade-offs](https://mlsanity.com/paper/2304.10819/) (2023)
- [On the Importance of Clinical Notes in Multi-modal Learning for EHR Data](https://mlsanity.com/paper/2212.03044/) (2022)
- [Why Chain of Thought Fails in Clinical Text Understanding](https://mlsanity.com/paper/2509.21933/) (2025)
- [Generalist embedding models are better at short-context clinical semantic search than specialized embedding models](https://mlsanity.com/paper/2401.01943/) (2024)
- [Detect and Classify -- Joint Span Detection and Classification for Health Outcomes](https://mlsanity.com/paper/2104.07789/) (2021)
- [What Do You See in this Patient? Behavioral Testing of Clinical NLP Models](https://mlsanity.com/paper/2111.15512/) (2021)
- [This Patient Looks Like That Patient: Prototypical Networks for Interpretable Diagnosis Prediction from Clinical Text](https://mlsanity.com/paper/2210.08500/) (2022)
- [Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints](https://mlsanity.com/paper/2210.09440/) (2022)
- [Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoning](https://mlsanity.com/paper/2301.08913/) (2023)
- [Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks](https://mlsanity.com/paper/2402.10597/) (2024)
- [Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences](https://mlsanity.com/paper/2502.20478/) (2025)
- [Quantifying Explainability in NLP and Analyzing Algorithms for Performance-Explainability Tradeoff](https://mlsanity.com/paper/2107.05693/) (2021)
- [BLIAM: Literature-based Data Synthesis for Synergistic Drug Combination Prediction](https://mlsanity.com/paper/2302.06860/) (2023)
- [CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment](https://mlsanity.com/paper/2502.06252/) (2025)
- [Patient Outcome and Zero-shot Diagnosis Prediction with Hypernetwork-guided Multitask Learning](https://mlsanity.com/paper/2109.03062/) (2021)
- [Assessing gender bias in medical and scientific masked language models with StereoSet](https://mlsanity.com/paper/2111.08088/) (2021)
- [Assessment of contextualised representations in detecting outcome phrases in clinical trials](https://mlsanity.com/paper/2203.03547/) (2022)
- [Cross-Lingual Knowledge Transfer for Clinical Phenotyping](https://mlsanity.com/paper/2208.01912/) (2022)
- [Soft-prompt tuning to predict lung cancer using primary care free-text Dutch medical notes](https://mlsanity.com/paper/2303.15846/) (2023)
- [DeVisE: Behavioral Testing of Medical Large Language Models](https://mlsanity.com/paper/2506.15339/) (2025)

## Abstract (arXiv)

Outcome prediction from clinical text can prevent doctors from overlooking possible risks and help hospitals to plan capacities. We simulate patients at admission time, when decision support can be especially valuable, and contribute a novel admission to discharge task with four common outcome prediction targets: Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction. The ideal system should infer outcomes based on symptoms, pre-conditions and risk factors of a patient. We evaluate the effectiveness of language models to handle this scenario and propose clinical outcome pre-training to integrate knowledge about patient outcomes from multiple public sources. We further present a simple method to incorporate ICD code hierarchy into the models. We show that our approach improves performance on the outcome tasks against several baselines. A detailed analysis reveals further strengths of the model, including transferability, but also weaknesses such as handling of vital values and inconsistencies in the underlying data.