mlsanityData to 14 Oct 2026

Lightweight Transformers for Clinical Natural Language Processing

arXiv 2302.04725 ↗First posted 9 Feb 2023cs.CL cs.AI cs.LGcited by 15

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Specialised pre-trained language models are becoming more frequent in NLP since they can potentially outperform models trained on generic texts. BioBERT and BioClinicalBERT are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like Knowledge Distillation (KD), it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. progress notes, discharge summaries etc). We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from 15 million to 65 million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including Natural Language Inference, Relation Extraction, Named Entity Recognition, and Sequence Classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https://huggingface.co/nlpie and Github page at https://github.com/nlpie-research/Lightweight-Clinical-Transformers, respectively, promoting reproducibility of our results.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 29 of its 70 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

29 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022On the Effectiveness of Compact Biomedical Transformers7 Sep

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
16 more
  1. 2020Efficient Transformers: A Surveycited by 790
  2. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  3. 2019Green AIcited by 627
  4. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  5. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  6. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  8. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  9. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  10. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  11. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  12. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  13. 2019Enhancing Clinical Concept Extraction with Contextual Embeddingscited by 48
  14. 2019Towards Automatic Generation of Shareable Synthetic Clinical Notes Using Neural Language Modelscited by 20
  15. 2021Extracting Adverse Drug Events from Clinical Notescited by 7
  16. 2022MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformerscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Real-Time Multimodal Cognitive Assistant for Emergency Medical Servicescited by 8
  2. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  3. 2024Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Taskscited by 4
  4. 2024Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillationcited by 3
  5. 2025A Novel Multi-Task Teacher-Student Architecture with Self-Supervised Pretraining for 48-Hour Vasoactive-Inotropic Trend Analysis in Sepsis Mortality Predictioncited by 3
  6. 2025Rethinking Tokenization for Clinical Time Series: When Less is Morecited by 3
  7. 2024CTP-LLM: Clinical Trial Phase Transition Prediction Using Large Language Modelscited by 2
  8. 2023Privacy Aware Question-Answering System for Online Mental Health Risk Assessmentcited by 1
  9. 2025Towards Scalable and Cross-Lingual Specialist Language Models for Oncologycited by 1
  10. 2025Early Warning Index for Patient Deteriorations in Hospitalscited by 1
  11. 2025A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reportscited by 0
  12. 2025MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Predictioncited by 0
3 more
  1. 2025ADMEDTAGGER: an annotation framework for distillation of expert knowledge for the Polish medical languagecited by 0
  2. 2026Resource-Conscious Modeling for Next- Day Discharge Prediction Using Clinical Notescited by 0
  3. 2026ESICA: A Scalable Framework for Text-Guided 3D Medical Image Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Distilling Large Language Models for Efficient Clinical Information Extraction
  2. 2023Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events
  3. 2024ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes
  4. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing
  5. 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insights
  6. 2024Large Language Models Struggle in Token-Level Clinical Named Entity Recognition
  7. 2024DSG-KD: Knowledge Distillation from Domain-Specific to General Language Models
  8. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation