# Lightweight Transformers for Clinical Natural Language Processing

arXiv: https://arxiv.org/abs/2302.04725 · HTML: https://mlsanity.com/paper/2302.04725/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 70 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Enhancing Clinical Concept Extraction with Contextual Embeddings](https://mlsanity.com/paper/1902.08691/) (2019)
- [Towards Automatic Generation of Shareable Synthetic Clinical Notes Using Neural Language Models](https://mlsanity.com/paper/1905.07002/) (2019)
- [On the Effectiveness of Compact Biomedical Transformers](https://mlsanity.com/paper/2209.03182/) (2022)
- [Extracting Adverse Drug Events from Clinical Notes](https://mlsanity.com/paper/2104.10791/) (2021)
- [MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers](https://mlsanity.com/paper/2210.06425/) (2022)

## Built on by (most cited first, 15 of 15)

- [Real-Time Multimodal Cognitive Assistant for Emergency Medical Services](https://mlsanity.com/paper/2403.06734/) (2024)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks](https://mlsanity.com/paper/2402.10597/) (2024)
- [Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillation](https://mlsanity.com/paper/2407.19540/) (2024)
- [A Novel Multi-Task Teacher-Student Architecture with Self-Supervised Pretraining for 48-Hour Vasoactive-Inotropic Trend Analysis in Sepsis Mortality Prediction](https://mlsanity.com/paper/2502.16834/) (2025)
- [Rethinking Tokenization for Clinical Time Series: When Less is More](https://mlsanity.com/paper/2512.05217/) (2025)
- [CTP-LLM: Clinical Trial Phase Transition Prediction Using Large Language Models](https://mlsanity.com/paper/2408.10995/) (2024)
- [Privacy Aware Question-Answering System for Online Mental Health Risk Assessment](https://mlsanity.com/paper/2306.05652/) (2023)
- [Towards Scalable and Cross-Lingual Specialist Language Models for Oncology](https://mlsanity.com/paper/2503.08323/) (2025)
- [Early Warning Index for Patient Deteriorations in Hospitals](https://mlsanity.com/paper/2512.14683/) (2025)
- [A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports](https://mlsanity.com/paper/2508.13796/) (2025)
- [MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction](https://mlsanity.com/paper/2508.19319/) (2025)
- [ADMEDTAGGER: an annotation framework for distillation of expert knowledge for the Polish medical language](https://mlsanity.com/paper/2601.09722/) (2025)
- [Resource-Conscious Modeling for Next- Day Discharge Prediction Using Clinical Notes](https://mlsanity.com/paper/2604.03498/) (2026)
- [ESICA: A Scalable Framework for Text-Guided 3D Medical Image Segmentation](https://mlsanity.com/paper/2604.24876/) (2026)

## Abstract (arXiv)

Specialised pre-trained language models are becoming more frequent in NLP since they can potentially outperform models trained on generic texts. BioBERT and BioClinicalBERT are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like Knowledge Distillation (KD), it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. progress notes, discharge summaries etc). We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from 15 million to 65 million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including Natural Language Inference, Relation Extraction, Named Entity Recognition, and Sequence Classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https://huggingface.co/nlpie and Github page at https://github.com/nlpie-research/Lightweight-Clinical-Transformers, respectively, promoting reproducibility of our results.