# An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain

arXiv: https://arxiv.org/abs/2012.15419 · HTML: https://mlsanity.com/paper/2012.15419/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 3 of 3)

- [Doctor XAvIer: Explainable Diagnosis on Physician-Patient Dialogues and XAI Evaluation](https://mlsanity.com/paper/2204.10178/) (2022)
- [Advancements in Natural Language Processing for Automatic Text Summarization](https://mlsanity.com/paper/2502.19773/) (2025)
- [Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition](https://mlsanity.com/paper/2310.13015/) (2023)

## Abstract (arXiv)

With the growing amount of text in health data, there have been rapid advances in large pre-trained models that can be applied to a wide variety of biomedical tasks with minimal task-specific modifications. Emphasizing the cost of these models, which renders technical replication challenging, this paper summarizes experiments conducted in replicating BioBERT and further pre-training and careful fine-tuning in the biomedical domain. We also investigate the effectiveness of domain-specific and domain-agnostic pre-trained models across downstream biomedical NLP tasks. Our finding confirms that pre-trained models can be impactful in some downstream NLP tasks (QA and NER) in the biomedical domain; however, this improvement may not justify the high cost of domain-specific pre-training.