# ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

arXiv: https://arxiv.org/abs/1904.05342 · HTML: https://mlsanity.com/paper/1904.05342/

## Key facts

- Cited by 493 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2019.

## Built on by (most cited first, 30 of 493)

- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains](https://mlsanity.com/paper/2210.04133/) (2022)
- [MS2: Multi-Document Summarization of Medical Studies](https://mlsanity.com/paper/2104.06486/) (2021)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [MultiLegalPile: A 689GB Multilingual Legal Corpus](https://mlsanity.com/paper/2306.02069/) (2023)
- [LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models](https://mlsanity.com/paper/2306.12420/) (2023)
- [Empowering Time Series Analysis with Large Language Models: A Survey](https://mlsanity.com/paper/2402.03182/) (2024)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://mlsanity.com/paper/2205.00619/) (2022)
- [HINT: Hierarchical Interaction Network for Trial Outcome Prediction Leveraging Web Data](https://mlsanity.com/paper/2102.04252/) (2021)
- [EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce](https://mlsanity.com/paper/2308.06966/) (2023)
- [PanGu-Coder: Program Synthesis with Function-Level Language Modeling](https://mlsanity.com/paper/2207.11280/) (2022)

## Abstract (arXiv)

Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intensive care unit. Code and model parameters are available.