mlsanityData to 7 Oct 2026

Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences

arXiv 2201.11838 ↗First posted 27 Jan 2022cs.CL cs.AIcited by 67

Not on the map. Why?

Abstract (arXiv)

Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. To overcome this, long sequence transformer models, e.g. Longformer and BigBird, were proposed with the idea of sparse attention mechanism to reduce the memory usage from quadratic to the sequence length to a linear scale. These models extended the maximum input sequence length from 512 to 4096, which enhanced the ability of modeling long-term dependency and consequently achieved optimal results in a variety of tasks. Inspired by the success of these long sequence transformer models, we introduce two domain enriched language models, namely Clinical-Longformer and Clinical-BigBird, which are pre-trained from large-scale clinical corpora. We evaluate both pre-trained models using 10 baseline tasks including named entity recognition, question answering, and document classification tasks. The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT as well as other short-sequence transformers in all downstream tasks. We have made our source code available at [https://github.com/luoyuanlab/Clinical-Longformer] the pre-trained models available for public download at: [https://huggingface.co/yikuan8/Clinical-Longformer].

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

28 of 46 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019Publicly Available Clinical BERT Embeddingscited by 995
  11. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  12. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
16 more
  1. 2018Graph Convolutional Networks for Text Classificationcited by 345
  2. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  3. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
  4. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  5. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  6. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  7. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  8. 2020BERT-XML: Large Scale Automated ICD Coding Using BERT Pretrainingcited by 37
  9. 2020Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognitioncited by 35
  10. 2018Cross-type Biomedical Named Entity Recognition with Deep Multi-Task Learningcited by 28
  11. 2020Clinical Reading Comprehension: A Thorough Analysis of the emrQA Datasetcited by 23
  12. 2020Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptationcited by 14
  13. 2018Early Prediction of Acute Kidney Injury in Critical Care Setting Using Clinical Notescited by 11
  14. 2018CollaboNet: collaboration of deep neural networks for biomedical named entity recognitioncited by 10
  15. 2019Adapting and evaluating a deep learning language model for clinical why-question answeringcited by 5
  16. 2021CancerBERT: a BERT model for Extracting Breast Cancer Phenotypes from Electronic Health Recordscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 15; 2024: 19; 2025: 17; 2026: 7.

67 papers.

Sort:

By area:

  1. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  2. 2023Do We Still Need Clinical Language Models?cited by 97
  3. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  4. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  5. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  6. 2023Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domaincited by 38
  7. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  8. 2023CPLLM: Clinical Prediction with Large Language Modelscited by 34
  9. 2022Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modelingcited by 33
  10. 2023Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extractioncited by 32
  11. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 26
  12. 2023INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosiscited by 24
38 more
  1. 2024ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notescited by 22
  2. 2022Gendered Mental Health Stigma in Masked Language Modelscited by 17
  3. 2022DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processingcited by 14
  4. 2022BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?cited by 13
  5. 2023Hierarchical Pretraining on Multimodal Electronic Health Recordscited by 11
  6. 2025LLMCARE: early detection of cognitive impairment via transformer models enhanced by LLM-generated synthetic datacited by 8
  7. 2024Generalist embedding models are better at short-context clinical semantic search than specialized embedding modelscited by 7
  8. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  9. 2024Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformercited by 6
  10. 2022TabText: Language-Based Representations of Tabular Health Data for Predictive Modellingcited by 5
  11. 2022Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Modelscited by 5
  12. 2024Classifying Cancer Stage with Open-Source Clinical Large Language Modelscited by 5
  13. 2024MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Trainingcited by 5
  14. 2022Extend and Explain: Interpreting Very Long Language Modelscited by 4
  15. 2024Global Contrastive Training for Multimodal Electronic Health Records with Language Supervisioncited by 4
  16. 2024Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Modelscited by 4
  17. 2025Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approachcited by 4
  18. 2023Progress Note Understanding -- Assessment and Plan Reasoning: Overview of the 2022 N2C2 Track 3 Shared Taskcited by 3
  19. 2023LongBoX: Evaluating Transformers on Long-Sequence Clinical Taskscited by 3
  20. 2024Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Recordscited by 3
  21. 2024FAIIR: Building Toward A Conversational AI Agent Assistant for Youth Mental Health Service Provisioncited by 3
  22. 2022An Automatic ICD Coding Network Using Partition-Based Label Attentioncited by 2
  23. 2023Automatic Personalized Impression Generation for PET Reports Using Large Language Modelscited by 2
  24. 2023CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviewscited by 2
  25. 2025SECRET: Semi-supervised Clinical Trial Document Similarity Searchcited by 2
  26. 2023DARE: Towards Robust Text Explanations in Biomedical and Healthcare Applicationscited by 1
  27. 2023Making the Most Out of the Limited Context Length: Predictive Power Varies with Clinical Note Type and Note Sectioncited by 1
  28. 2024Bespoke Large Language Models for Digital Triage Assistance in Mental Health Carecited by 1
  29. 2024Summarizing Radiology Reports Findings into Impressionscited by 1
  30. 2024Tumor Location-weighted MRI-Report Contrastive Learning: A Framework for Improving the Explainability of Pediatric Brain Tumor Diagnosiscited by 1
  31. 2024NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Textcited by 1
  32. 2025Med-gte-hybrid: A contextual embedding transformer model for extracting actionable information from clinical textscited by 1
  33. 2025Comparing representations of long clinical texts for the task of patient note-identificationcited by 1
  34. 2025Bridging Electronic Health Records and Clinical Texts: Contrastive Learning for Enhanced Clinical Taskscited by 1
  35. 2025Knowledge Elicitation with Large Language Models for Interpretable Cancer Stage Identification from Pathology Reportscited by 1
  36. 2025SurvBench: A Standardised Preprocessing Pipeline for Multi-Modal Electronic Health Record Survival Analysiscited by 1
  37. 2025Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attributioncited by 1
  38. 2023On Preserving the Knowledge of Long Clinical Textscited by 0

50 most cited of 67 All 67 (JSON)