# Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies arXiv: https://arxiv.org/abs/2102.08366 ยท HTML: https://mlsanity.com/paper/2102.08366/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 20 references that are in the map. - First posted on arXiv 16 Feb 2021. ## Builds on (most cited first, 18 of 18) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016) - [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019) - [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019) - [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019) - [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019) - [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019) - [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020) - [Construction of the Literature Graph in Semantic Scholar](https://mlsanity.com/paper/1805.02262/) (2018) - [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020) - [Pre-trained Language Model for Biomedical Question Answering](https://mlsanity.com/paper/1909.08229/) (2019) ## Built on by (most cited first, 21 of 21) - [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023) - [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023) - [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021) - [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021) - [Summarizing Patients Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models](https://mlsanity.com/paper/2208.08408/) (2022) - [NapSS: Paragraph-level Medical Text Simplification via Narrative Prompting and Sentence-matching Summarization](https://mlsanity.com/paper/2302.05574/) (2023) - [Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study](https://mlsanity.com/paper/2402.15663/) (2024) - [Event Temporal Relation Extraction with Bayesian Translational Model](https://mlsanity.com/paper/2302.04985/) (2023) - [Prompting-based Synthetic Data Generation for Few-Shot Question Answering](https://mlsanity.com/paper/2405.09335/) (2024) - [Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation](https://mlsanity.com/paper/2210.12902/) (2022) - [Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords](https://mlsanity.com/paper/2307.07160/) (2023) - [Disentangled Learning of Stance and Aspect Topics for Vaccine Attitude Detection in Social Media](https://mlsanity.com/paper/2205.03296/) (2022) - [BioADAPT-MRC: Adversarial Learning-based Domain Adaptation Improves Biomedical Machine Reading Comprehension Task](https://mlsanity.com/paper/2202.13174/) (2022) - [Question answering systems for health professionals at the point of care -- a systematic review](https://mlsanity.com/paper/2402.01700/) (2024) - [Cascading Large Language Models for Salient Event Graph Generation](https://mlsanity.com/paper/2406.18449/) (2024) - [SciGisPy: a Novel Metric for Biomedical Text Simplification via Gist Inference Score](https://mlsanity.com/paper/2410.09632/) (2024) - [Hero-Gang Neural Model For Named Entity Recognition](https://mlsanity.com/paper/2205.07177/) (2022) - [Building Efficient and Effective OpenQA Systems for Low-Resource Languages](https://mlsanity.com/paper/2401.03590/) (2024) - [Society of Medical Simplifiers](https://mlsanity.com/paper/2410.09631/) (2024) - [Robust Biomedical Publication Type and Study Design Classification with Knowledge-Guided Perturbations](https://mlsanity.com/paper/2605.11502/) (2026) - [DIRECT: Direct Decoding for Efficient and Aligned Sequence Labeling with Large Language Models](https://mlsanity.com/paper/2607.26891/) (2026) ## Abstract (arXiv) Biomedical question-answering (QA) has gained increased attention for its capability to provide users with high-quality information from a vast scientific literature. Although an increasing number of biomedical QA datasets has been recently made available, those resources are still rather limited and expensive to produce. Transfer learning via pre-trained language models (LMs) has been shown as a promising approach to leverage existing general-purpose knowledge. However, finetuning these large models can be costly and time consuming, often yielding limited benefits when adapting to specific themes of specialised domains, such as the COVID-19 literature. To bootstrap further their domain adaptation, we propose a simple yet unexplored approach, which we call biomedical entity-aware masking (BEM). We encourage masked language models to learn entity-centric knowledge based on the pivotal entities characterizing the domain at hand, and employ those entities to drive the LM fine-tuning. The resulting strategy is a downstream process applicable to a wide variety of masked LMs, not requiring additional memory or components in the neural architectures. Experimental results show performance on par with state-of-the-art models on several biomedical QA datasets.