# UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus

arXiv: https://arxiv.org/abs/2010.10391 · HTML: https://mlsanity.com/paper/2010.10391/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 30 of 53)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model](https://mlsanity.com/paper/2304.03086/) (2023)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques](https://mlsanity.com/paper/2403.05881/) (2024)
- [Improving Biomedical Pretrained Language Models with Knowledge](https://mlsanity.com/paper/2104.10344/) (2021)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2110.08173/) (2021)
- [RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records](https://mlsanity.com/paper/2403.00815/) (2024)
- [Literature-Augmented Clinical Outcome Prediction](https://mlsanity.com/paper/2111.08374/) (2021)
- [MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Score](https://mlsanity.com/paper/2210.05875/) (2022)
- [Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinese](https://mlsanity.com/paper/2309.04175/) (2023)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Probing Pre-Trained Language Models for Disease Knowledge](https://mlsanity.com/paper/2106.07285/) (2021)
- [Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarios](https://mlsanity.com/paper/2410.11114/) (2024)
- [Clinical Relation Extraction Using Transformer-based Models](https://mlsanity.com/paper/2107.08957/) (2021)
- [Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports](https://mlsanity.com/paper/2304.13180/) (2023)
- [Developing a general-purpose clinical language inference model from a large corpus of clinical notes](https://mlsanity.com/paper/2210.06566/) (2022)
- [Position-based Prompting for Health Outcome Generation](https://mlsanity.com/paper/2204.03489/) (2022)
- [Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words](https://mlsanity.com/paper/2209.06453/) (2022)
- [Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning](https://mlsanity.com/paper/2505.17436/) (2025)
- [A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts](https://mlsanity.com/paper/2101.10196/) (2021)
- [Coreference Resolution for the Biomedical Domain: A Survey](https://mlsanity.com/paper/2109.12424/) (2021)
- [Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction](https://mlsanity.com/paper/2204.10360/) (2022)
- [Representation learning to advance multi-institutional studies with electronic health record data from US and France](https://mlsanity.com/paper/2502.08547/) (2025)
- [Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences](https://mlsanity.com/paper/2502.20478/) (2025)
- [Should Semantic Vector Composition be Explicit? Can it be Linear?](https://mlsanity.com/paper/2104.06555/) (2021)
- [Review of Natural Language Processing in Pharmacology](https://mlsanity.com/paper/2208.10228/) (2022)

## Abstract (arXiv)

Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration expert domain knowledge. In this work, we introduced UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was performed in two ways: i) connecting words that have the same underlying `concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to create clinically meaningful input embeddings. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models on common named-entity recognition (NER) and clinical natural language inference clinical NLP tasks.