# The Utility of General Domain Transfer Learning for Medical Language Tasks

arXiv: https://arxiv.org/abs/2002.06670 · HTML: https://mlsanity.com/paper/2002.06670/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2020.

## Built on by (most cited first, 2 of 2)

- [Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network](https://mlsanity.com/paper/2210.00881/) (2022)
- [An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotyping](https://mlsanity.com/paper/2311.08687/) (2023)

## Abstract (arXiv)

The purpose of this study is to analyze the efficacy of transfer learning techniques and transformer-based models as applied to medical natural language processing (NLP) tasks, specifically radiological text classification. We used 1,977 labeled head CT reports, from a corpus of 96,303 total reports, to evaluate the efficacy of pretraining using general domain corpora and a combined general and medical domain corpus with a bidirectional representations from transformers (BERT) model for the purpose of radiological text classification. Model performance was benchmarked to a logistic regression using bag-of-words vectorization and a long short-term memory (LSTM) multi-label multi-class classification model, and compared to the published literature in medical text classification. The BERT models using either set of pretrained checkpoints outperformed the logistic regression model, achieving sample-weighted average F1-scores of 0.87 and 0.87 for the general domain model and the combined general and biomedical-domain model. General text transfer learning may be a viable technique to generate state-of-the-art results within medical NLP tasks on radiological corpora, outperforming other deep models such as LSTMs. The efficacy of pretraining and transformer-based models could serve to facilitate the creation of groundbreaking NLP models in the uniquely challenging data environment of medical text.