# CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain arXiv: https://arxiv.org/abs/2112.08754 ยท HTML: https://mlsanity.com/paper/2112.08754/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 44 references that are in the map. - First posted on arXiv 16 Dec 2021. ## Builds on (most cited first, 22 of 22) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019) - [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019) - [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019) - [FLERT: Document-Level Features for Named Entity Recognition](https://mlsanity.com/paper/2011.06993/) (2020) - [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020) - [Exploring Cross-sentence Contexts for Named Entity Recognition with BERT](https://mlsanity.com/paper/2006.01563/) (2020) - [HunFlair: An Easy-to-Use Tool for State-of-the-Art Biomedical Named Entity Recognition](https://mlsanity.com/paper/2008.07347/) (2020) - [Few-shot Learning for Named Entity Recognition in Medical Text](https://mlsanity.com/paper/1811.05468/) (2018) - [BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition](https://mlsanity.com/paper/2009.09223/) (2020) - [MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning](https://mlsanity.com/paper/2004.10220/) (2020) - [Transfer Learning for Named-Entity Recognition with Neural Networks](https://mlsanity.com/paper/1705.06273/) (2017) - [MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers](https://mlsanity.com/paper/2001.08904/) (2020) - [To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning](https://mlsanity.com/paper/2104.08078/) (2021) - [NLNDE: The Neither-Language-Nor-Domain-Experts' Way of Spanish Medical Document De-Identification](https://mlsanity.com/paper/2007.01030/) (2020) - [NLNDE at CANTEMIST: Neural Sequence Labeling and Parsing Approaches for Clinical Concept Extraction](https://mlsanity.com/paper/2010.12322/) (2020) - [Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Setting](https://mlsanity.com/paper/2109.08597/) (2021) ## Built on by (most cited first, 7 of 7) - [SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains](https://mlsanity.com/paper/2302.06868/) (2023) - [NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis](https://mlsanity.com/paper/2305.00090/) (2023) - [Causality-aware Concept Extraction based on Knowledge-guided Prompting](https://mlsanity.com/paper/2305.01876/) (2023) - [TADA: Efficient Task-Agnostic Domain Adaptation for Transformers](https://mlsanity.com/paper/2305.12717/) (2023) - [Concept-Oriented Deep Learning with Large Language Models](https://mlsanity.com/paper/2306.17089/) (2023) - [Multi-lingual Multi-institutional Electronic Health Record based Predictive Model](https://mlsanity.com/paper/2604.00027/) (2026) - [Team Fusion@ SU@ BC8 SympTEMIST track: transformer-based approach for symptom recognition and linking](https://mlsanity.com/paper/2604.06424/) (2026) ## Abstract (arXiv) The field of natural language processing (NLP) has recently seen a large change towards using pre-trained language models for solving almost any task. Despite showing great improvements in benchmark datasets for various tasks, these models often perform sub-optimal in non-standard domains like the clinical domain where a large gap between pre-training documents and target documents is observed. In this paper, we aim at closing this gap with domain-specific training of the language model and we investigate its effect on a diverse set of downstream tasks and settings. We introduce the pre-trained CLIN-X (Clinical XLM-R) language models and show how CLIN-X outperforms other pre-trained transformer models by a large margin for ten clinical concept extraction tasks from two languages. In addition, we demonstrate how the transformer model can be further improved with our proposed task- and language-agnostic model architecture based on ensembles over random splits and cross-sentence context. Our studies in low-resource and transfer settings reveal stable model performance despite a lack of annotated data with improvements of up to 47 F1 points when only 250 labeled sentences are available. Our results highlight the importance of specialized language models as CLIN-X for concept extraction in non-standard domains, but also show that our task-agnostic model architecture is robust across the tested tasks and languages so that domain- or task-specific adaptations are not required.