# Enhancing Clinical Concept Extraction with Contextual Embeddings

arXiv: https://arxiv.org/abs/1902.08691 · HTML: https://mlsanity.com/paper/1902.08691/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2019.

## Built on by (most cited first, 30 of 48)

- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [Signature Methods in Machine Learning](https://mlsanity.com/paper/2206.14674/) (2022)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Importance-Aware Adaptive Dataset Distillation](https://mlsanity.com/paper/2401.15863/) (2024)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Lightweight Transformers for Clinical Natural Language Processing](https://mlsanity.com/paper/2302.04725/) (2023)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Data Augmentations for Improved (Large) Language Model Generalization](https://mlsanity.com/paper/2310.12803/) (2023)
- [NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs](https://mlsanity.com/paper/2312.11803/) (2023)
- [Clinical Relation Extraction Using Transformer-based Models](https://mlsanity.com/paper/2107.08957/) (2021)
- [GersteinLab at MEDIQA-Chat 2023: Clinical Note Summarization from Doctor-Patient Conversations through Fine-tuning and In-context Learning](https://mlsanity.com/paper/2305.05001/) (2023)
- [Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?](https://mlsanity.com/paper/2411.10020/) (2024)
- [Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification](https://mlsanity.com/paper/2104.08444/) (2021)
- [Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature](https://mlsanity.com/paper/2106.13375/) (2021)
- [Generative Large Language Models Are All-purpose Text Analytics Engines: Text-to-text Learning Is All Your Need](https://mlsanity.com/paper/2312.06099/) (2023)
- [Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers](https://mlsanity.com/paper/2411.01645/) (2024)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [JaMIE: A Pipeline Japanese Medical Information Extraction System](https://mlsanity.com/paper/2111.04261/) (2021)
- [Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approach](https://mlsanity.com/paper/2503.18085/) (2025)
- [Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings](https://mlsanity.com/paper/2505.21242/) (2025)
- [Human-interpretable clustering of short-text using large language models](https://mlsanity.com/paper/2405.07278/) (2024)
- [Concept Identification of Directly and Indirectly Related Mentions Referring to Groups of Persons](https://mlsanity.com/paper/2107.00955/) (2021)
- [Neural Medication Extraction: A Comparison of Recent Models in Supervised and Semi-supervised Learning Settings](https://mlsanity.com/paper/2110.10213/) (2021)
- [Fine-Tuning BERT for Automatic ADME Semantic Labeling in FDA Drug Labeling to Enhance Product-Specific Guidance Assessment](https://mlsanity.com/paper/2207.12376/) (2022)
- [NEAR: Named Entity and Attribute Recognition of clinical concepts](https://mlsanity.com/paper/2208.13949/) (2022)
- [A Pre-trained Data Deduplication Model based on Active Learning](https://mlsanity.com/paper/2308.00721/) (2023)
- [The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through](https://mlsanity.com/paper/2203.15364/) (2022)
- [AKI-BERT: a Pre-trained Clinical Language Model for Early Prediction of Acute Kidney Injury](https://mlsanity.com/paper/2205.03695/) (2022)
- [Word-Graph2vec: An efficient word embedding approach on word co-occurrence graph using random walk technique](https://mlsanity.com/paper/2301.04312/) (2023)

## Abstract (arXiv)

Neural network-based representations ("embeddings") have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (e.g., ELMo, BERT) have further pushed the state-of-the-art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). Both off-the-shelf open-domain embeddings and pre-trained clinical embeddings from MIMIC-III are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings, and compare these on four concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pre-training time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings. Contextual embeddings pre-trained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65. We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate contextual embeddings encode valuable semantic information not accounted for in traditional word representations.