mlsanityData to 7 Oct 2026

Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA

arXiv 2004.03354 ↗First posted 7 Apr 2020cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Domain adaptation of Pretrained Language Models (PTLMs) is typically achieved by unsupervised pretraining on target-domain text. While successful, this approach is expensive in terms of hardware, runtime and CO_2 emissions. Here, we propose a cheaper alternative: We train Word2Vec on target-domain text and align the resulting word vectors with the wordpiece vectors of a general-domain PTLM. We evaluate on eight biomedical Named Entity Recognition (NER) tasks and compare against the recently proposed BioBERT model. We cover over 60% of the BioBERT-BERT F1 delta, at 5% of BioBERT's CO_2 footprint and 2% of its cloud compute cost. We also show how to quickly adapt an existing general-domain Question Answering (QA) model to an emerging domain: the Covid-19 pandemic.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 1; 2023: 3; 2024: 4; 2025: 4.

23 papers.

Sort:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  4. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  5. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  6. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  7. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  8. 2021AVocaDo: Strategy for Adapting Vocabulary to Downstream Domaincited by 20
  9. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  10. 2021Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERTcited by 14
  11. 2021MDAPT: Multilingual Domain Adaptive Pretraining in a Single Modelcited by 6
  12. 2023Improving Small Language Models on PubMedQA via Generative Data Augmentationcited by 6
11 more
  1. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  2. 2021Task-adaptive Pre-training of Language Models with Word Embedding Regularizationcited by 5
  3. 2021On the Universality of Deep Contextual Language Modelscited by 4
  4. 2025Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrievalcited by 4
  5. 2024Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rustcited by 3
  6. 2023BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognitioncited by 2
  7. 2024Extracting Biomedical Entities from Noisy Audio Transcriptscited by 1
  8. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  9. 2021Improving Span Representation for Domain-adapted Coreference Resolutioncited by 0
  10. 2024Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NERcited by 0
  11. 2025MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learningcited by 0