# Unsupervised Pre-training for Biomedical Question Answering

arXiv: https://arxiv.org/abs/2009.12952 · HTML: https://mlsanity.com/paper/2009.12952/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [BioADAPT-MRC: Adversarial Learning-based Domain Adaptation Improves Biomedical Machine Reading Comprehension Task](https://mlsanity.com/paper/2202.13174/) (2022)
- [Now You See Me: Robust approach to Partial Occlusions](https://mlsanity.com/paper/2304.11779/) (2023)
- [Text2Time: Transformer-based Article Time Period Prediction](https://mlsanity.com/paper/2304.10859/) (2023)
- [Ultra Sharp : Study of Single Image Super Resolution using Residual Dense Network](https://mlsanity.com/paper/2304.10870/) (2023)

## Abstract (arXiv)

We explore the suitability of unsupervised representation learning methods on biomedical text -- BioBERT, SciBERT, and BioSentVec -- for biomedical question answering. To further improve unsupervised representations for biomedical QA, we introduce a new pre-training task from unlabeled data designed to reason about biomedical entities in the context. Our pre-training method consists of corrupting a given context by randomly replacing some mention of a biomedical entity with a random entity mention and then querying the model with the correct entity mention in order to locate the corrupted part of the context. This de-noising task enables the model to learn good representations from abundant, unlabeled biomedical text that helps QA tasks and minimizes the train-test mismatch between the pre-training task and the downstream QA tasks by requiring the model to predict spans. Our experiments show that pre-training BioBERT on the proposed pre-training task significantly boosts performance and outperforms the previous best model from the 7th BioASQ Task 7b-Phase B challenge.