# Pre-trained Language Model for Biomedical Question Answering

arXiv: https://arxiv.org/abs/1909.08229 · HTML: https://mlsanity.com/paper/1909.08229/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 30 of 31)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering](https://mlsanity.com/paper/2311.06503/) (2023)
- [Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing](https://mlsanity.com/paper/2203.00130/) (2022)
- [Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies](https://mlsanity.com/paper/2102.08366/) (2021)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Overview of BioASQ 2020: The eighth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering](https://mlsanity.com/paper/2106.14618/) (2021)
- [Sequence tagging for biomedical extractive question answering](https://mlsanity.com/paper/2104.07535/) (2021)
- [ELECTRAMed: a new pre-trained language representation model for biomedical NLP](https://mlsanity.com/paper/2104.09585/) (2021)
- [The Semantic Reader Project: Augmenting Scholarly Documents through AI-Powered Interactive Reading Interfaces](https://mlsanity.com/paper/2303.14334/) (2023)
- [Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes](https://mlsanity.com/paper/2302.05752/) (2023)
- [Detect and Classify -- Joint Span Detection and Classification for Health Outcomes](https://mlsanity.com/paper/2104.07789/) (2021)
- [Data Augmentation for Biomedical Factoid Question Answering](https://mlsanity.com/paper/2204.04711/) (2022)
- [Impact of Tokenization on LLaMa Russian Adaptation](https://mlsanity.com/paper/2312.02598/) (2023)
- [Knowledge of Pretrained Language Models on Surface Information of Tokens](https://mlsanity.com/paper/2402.09808/) (2024)
- [BioADAPT-MRC: Adversarial Learning-based Domain Adaptation Improves Biomedical Machine Reading Comprehension Task](https://mlsanity.com/paper/2202.13174/) (2022)
- [Pre-Trained Language Models for Keyphrase Prediction: A Review](https://mlsanity.com/paper/2409.01087/) (2024)
- [Augmenting Biomedical Named Entity Recognition with General-domain Resources](https://mlsanity.com/paper/2406.10671/) (2024)
- [Science Checker: Extractive-Boolean Question Answering For Scientific Fact Checking](https://mlsanity.com/paper/2204.12263/) (2022)
- [Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals](https://mlsanity.com/paper/2103.11011/) (2021)
- [Transformer-based Language Models for Factoid Question Answering at BioASQ9b](https://mlsanity.com/paper/2109.07185/) (2021)
- [Recent Advances in Automated Question Answering In Biomedical Domain](https://mlsanity.com/paper/2111.05937/) (2021)
- [Using Weak Supervision and Data Augmentation in Question Answering](https://mlsanity.com/paper/2309.16175/) (2023)
- [Fine-tuning pre-trained extractive QA models for clinical document parsing](https://mlsanity.com/paper/2312.02314/) (2023)
- [A Question Answering Based Pipeline for Comprehensive Chinese EHR Information Extraction](https://mlsanity.com/paper/2402.11177/) (2024)

## Abstract (arXiv)

The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.