# A Comparative Study of Pretrained Language Models for Long Clinical Text

arXiv: https://arxiv.org/abs/2301.11847 · HTML: https://mlsanity.com/paper/2301.11847/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 27 Jan 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus](https://mlsanity.com/paper/2010.10391/) (2020)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition](https://mlsanity.com/paper/2010.03746/) (2020)
- [Cross-type Biomedical Named Entity Recognition with Deep Multi-Task Learning](https://mlsanity.com/paper/1801.09851/) (2018)
- [Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset](https://mlsanity.com/paper/2005.00574/) (2020)
- [Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation](https://mlsanity.com/paper/2010.02705/) (2020)
- [Early Prediction of Acute Kidney Injury in Critical Care Setting Using Clinical Notes](https://mlsanity.com/paper/1811.02757/) (2018)
- [CollaboNet: collaboration of deep neural networks for biomedical named entity recognition](https://mlsanity.com/paper/1809.07950/) (2018)
- [Leveraging Time Irreversibility with Order-Contrastive Pre-training](https://mlsanity.com/paper/2111.02599/) (2021)
- [Adapting and evaluating a deep learning language model for clinical why-question answering](https://mlsanity.com/paper/1911.05604/) (2019)
- [CancerBERT: a BERT model for Extracting Breast Cancer Phenotypes from Electronic Health Records](https://mlsanity.com/paper/2108.11303/) (2021)

## Built on by (most cited first, 30 of 35)

- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.00036/) (2024)
- [Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation](https://mlsanity.com/paper/2405.19093/) (2024)
- [ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks](https://mlsanity.com/paper/2407.18525/) (2024)
- [Question Answering for Electronic Health Records: A Scoping Review of datasets and models](https://mlsanity.com/paper/2310.08759/) (2023)
- [MedConceptsQA: Open Source Medical Concepts QA Benchmark](https://mlsanity.com/paper/2405.07348/) (2024)
- [Uncertainty-aware abstention in medical diagnosis based on medical texts](https://mlsanity.com/paper/2502.18050/) (2025)
- [Length-Aware Multi-Kernel Transformer for Long Document Classification](https://mlsanity.com/paper/2405.07052/) (2024)
- [BELHD: Improving Biomedical Entity Linking with Homonoym Disambiguation](https://mlsanity.com/paper/2401.05125/) (2024)
- [Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models](https://mlsanity.com/paper/2412.17803/) (2024)
- [Auxiliary Knowledge-Induced Learning for Automatic Multi-Label Medical Document Classification](https://mlsanity.com/paper/2405.19084/) (2024)
- [Synthetic4Health: Generating Annotated Synthetic Clinical Letters](https://mlsanity.com/paper/2409.09501/) (2024)
- [Large Language Models for Healthcare Text Classification: A Systematic Review](https://mlsanity.com/paper/2503.01159/) (2025)
- [Clinical Note Owns its Hierarchy: Multi-Level Hypergraph Neural Networks for Patient-Level Representation Learning](https://mlsanity.com/paper/2305.09756/) (2023)
- [Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study](https://mlsanity.com/paper/2402.16689/) (2024)
- [Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting](https://mlsanity.com/paper/2403.13369/) (2024)
- [SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials](https://mlsanity.com/paper/2404.03977/) (2024)
- [Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings](https://mlsanity.com/paper/2406.16611/) (2024)
- [MedSlice: Fine-Tuned Large Language Models for Secure Clinical Note Sectioning](https://mlsanity.com/paper/2501.14105/) (2025)
- [Large Language Models for Granularized Barrett's Esophagus Diagnosis Classification](https://mlsanity.com/paper/2308.08660/) (2023)
- [Enhanced Labeling Technique for Reddit Text and Fine-Tuned Longformer Models for Classifying Depression Severity in English and Luganda](https://mlsanity.com/paper/2401.14240/) (2024)
- [Aspect-Oriented Summarization for Psychiatric Short-Term Readmission Prediction](https://mlsanity.com/paper/2502.10388/) (2025)
- [Med-gte-hybrid: A contextual embedding transformer model for extracting actionable information from clinical texts](https://mlsanity.com/paper/2502.15996/) (2025)
- [Comparing representations of long clinical texts for the task of patient note-identification](https://mlsanity.com/paper/2503.24006/) (2025)
- [Privacy-Preserving Generation of Clinical Narratives from Medical Terminologies](https://mlsanity.com/paper/2509.10882/) (2025)
- [Can-SAVE: Deploying Low-Cost and Population-Scale Cancer Screening via Survival Analysis Variables and EHR](https://mlsanity.com/paper/2309.15039/) (2023)
- [Zero-Shot Medical Information Retrieval via Knowledge Graph Embedding](https://mlsanity.com/paper/2310.20588/) (2023)
- [Towards Reducing Diagnostic Errors with Interpretable Risk Prediction](https://mlsanity.com/paper/2402.10109/) (2024)
- [Evaluating Large Language Models for Anxiety and Depression Classification using Counseling and Psychotherapy Transcripts](https://mlsanity.com/paper/2407.13228/) (2024)
- [Domain-specific long text classification from sparse relevant information](https://mlsanity.com/paper/2408.13253/) (2024)

## Abstract (arXiv)

Objective: Clinical knowledge enriched transformer models (e.g., ClinicalBERT) have state-of-the-art results on clinical NLP (natural language processing) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (e.g., Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts. Materials and Methods: Inspired by the success of long sequence transformer models and the fact that clinical notes are mostly long, we introduce two domain enriched language models, Clinical-Longformer and Clinical-BigBird, which are pre-trained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks. Results: The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results. Discussion: Our pre-trained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https://github.com/luoyuanlab/Clinical-Longformer, and the pre-trained models available for public download at: https://huggingface.co/yikuan8/Clinical-Longformer. Conclusion: This study demonstrates that clinical knowledge enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.