# Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation

arXiv: https://arxiv.org/abs/1912.11975 · HTML: https://mlsanity.com/paper/1912.11975/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2019.

## Built on by (most cited first, 30 of 30)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health](https://mlsanity.com/paper/2304.10447/) (2023)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Literature-Augmented Clinical Outcome Prediction](https://mlsanity.com/paper/2111.08374/) (2021)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [In-BoXBART: Get Instructions into Biomedical Multi-Task Learning](https://mlsanity.com/paper/2204.07600/) (2022)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)
- [Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding](https://mlsanity.com/paper/2204.10716/) (2022)
- [Estimating Redundancy in Clinical Text](https://mlsanity.com/paper/2105.11832/) (2021)
- [Clinical Relation Extraction Using Transformer-based Models](https://mlsanity.com/paper/2107.08957/) (2021)
- [MedINST: Meta Dataset of Biomedical Instructions](https://mlsanity.com/paper/2410.13458/) (2024)
- [Deep learning models are not robust against noise in clinical text](https://mlsanity.com/paper/2108.12242/) (2021)
- [Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints](https://mlsanity.com/paper/2210.09440/) (2022)
- [Deep Learning, Natural Language Processing, and Explainable Artificial Intelligence in the Biomedical Domain](https://mlsanity.com/paper/2202.12678/) (2022)
- [Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance](https://mlsanity.com/paper/2103.05841/) (2021)
- [Contrastive Learning on Multimodal Analysis of Electronic Health Records](https://mlsanity.com/paper/2403.14926/) (2024)
- [OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography](https://mlsanity.com/paper/2409.09052/) (2024)
- [Using Selective Masking as a Bridge between Pre-training and Fine-tuning](https://mlsanity.com/paper/2211.13815/) (2022)
- [Clinical Note Owns its Hierarchy: Multi-Level Hypergraph Neural Networks for Patient-Level Representation Learning](https://mlsanity.com/paper/2305.09756/) (2023)
- [An Empirical Analysis for Zero-Shot Multi-Label Classification on COVID-19 CT Scans and Uncurated Reports](https://mlsanity.com/paper/2309.01740/) (2023)
- [Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embedding](https://mlsanity.com/paper/2506.13104/) (2025)
- [CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction](https://mlsanity.com/paper/2511.11423/) (2025)
- [A Multi-View Joint Learning Framework for Embedding Clinical Codes and Text Using Graph Neural Networks](https://mlsanity.com/paper/2301.11608/) (2023)
- [Early Risk Prediction with Temporally and Contextually Grounded Clinical Language Processing](https://mlsanity.com/paper/2511.22038/) (2025)

## Abstract (arXiv)

Clinical notes contain rich data, which is unexploited in predictive modeling compared to structured data. In this work, we developed a new text representation Clinical XLNet for clinical notes which also leverages the temporal information of the sequence of the notes. We evaluated our models on prolonged mechanical ventilation prediction problem and our experiments demonstrated that Clinical XLNet outperforms the best baselines consistently.