# Clinical Risk Prediction Using Language Models: Benefits And Considerations

arXiv: https://arxiv.org/abs/2312.03742 · HTML: https://mlsanity.com/paper/2312.03742/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record](https://mlsanity.com/paper/1810.04793/) (2018)
- [Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Risk](https://mlsanity.com/paper/1905.08547/) (2019)
- [Self-supervised Pre-training for Semantic Segmentation in an Indoor Scene](https://mlsanity.com/paper/2210.01884/) (2022)
- [GenSyn: A Multi-stage Framework for Generating Synthetic Microdata using Macro Data Sources](https://mlsanity.com/paper/2212.05975/) (2022)

## Built on by (most cited first, 6 of 6)

- [Large Language Models are Powerful Electronic Health Record Encoders](https://mlsanity.com/paper/2502.17403/) (2025)
- [DeLLiriuM: A large language model for delirium prediction in the ICU using structured EHR](https://mlsanity.com/paper/2410.17363/) (2024)
- [Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints](https://mlsanity.com/paper/2604.01841/) (2026)
- [AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks](https://mlsanity.com/paper/2605.10286/) (2026)
- [PaReGTA: A Temporally Aware LLM-Based Patient Representation Framework for EHR Analytics](https://mlsanity.com/paper/2602.19661/) (2026)
- [A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Data](https://mlsanity.com/paper/2606.28798/) (2026)

## Abstract (arXiv)

The utilization of Electronic Health Records (EHRs) for clinical risk prediction is on the rise. However, strict privacy regulations limit access to comprehensive health records, making it challenging to apply standard machine learning algorithms in practical real-world scenarios. Previous research has addressed this data limitation by incorporating medical ontologies and employing transfer learning methods. In this study, we investigate the potential of leveraging language models (LMs) as a means to incorporate supplementary domain knowledge for improving the performance of various EHR-based risk prediction tasks. Unlike applying LMs to unstructured EHR data such as clinical notes, this study focuses on using textual descriptions within structured EHR to make predictions exclusively based on that information. We extensively compare against previous approaches across various data types and sizes. We find that employing LMs to represent structured EHRs, such as diagnostic histories, leads to improved or at least comparable performance in diverse risk prediction tasks. Furthermore, LM-based approaches offer numerous advantages, including few-shot learning, the capability to handle previously unseen medical concepts, and adaptability to various medical vocabularies. Nevertheless, we underscore, through various experiments, the importance of being cautious when employing such models, as concerns regarding the reliability of LMs persist.