# Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition

arXiv: https://arxiv.org/abs/2010.03746 · HTML: https://mlsanity.com/paper/2010.03746/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 30 of 35)

- [DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation](https://mlsanity.com/paper/2308.14346/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [Scientific Fact-Checking: A Survey of Resources and Approaches](https://mlsanity.com/paper/2305.16859/) (2023)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets](https://mlsanity.com/paper/2204.08997/) (2022)
- [ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model](https://mlsanity.com/paper/2304.03086/) (2023)
- [Improving Biomedical Pretrained Language Models with Knowledge](https://mlsanity.com/paper/2104.10344/) (2021)
- [A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks](https://mlsanity.com/paper/2310.04270/) (2023)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [Understanding Telecom Language Through Large Language Models](https://mlsanity.com/paper/2306.07933/) (2023)
- [Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models](https://mlsanity.com/paper/2305.19249/) (2023)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Cross-Domain Data Integration for Named Entity Disambiguation in Biomedical Text](https://mlsanity.com/paper/2110.08228/) (2021)
- [DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding](https://mlsanity.com/paper/2112.01047/) (2021)
- [Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT](https://mlsanity.com/paper/2109.04810/) (2021)
- [Hierarchical Annotation for Building A Suite of Clinical Natural Language Processing Tasks: Progress Note Understanding](https://mlsanity.com/paper/2204.03035/) (2022)
- [Probing Pre-Trained Language Models for Disease Knowledge](https://mlsanity.com/paper/2106.07285/) (2021)
- [Unsupervised pre-training of graph transformers on patient population graphs](https://mlsanity.com/paper/2207.10603/) (2022)
- [TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing](https://mlsanity.com/paper/2203.10839/) (2022)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)
- [CliMedBERT: A Pre-trained Language Model for Climate and Health-related Text](https://mlsanity.com/paper/2212.00689/) (2022)
- [Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision](https://mlsanity.com/paper/2209.15301/) (2022)
- [Unsupervised Knowledge Graph Construction and Event-centric Knowledge Infusion for Scientific NLI](https://mlsanity.com/paper/2210.15248/) (2022)
- [Generating multiple-choice questions for medical question answering with distractors and cue-masking](https://mlsanity.com/paper/2303.07069/) (2023)
- [Knowledge-Informed Machine Learning for Cancer Diagnosis and Prognosis: A review](https://mlsanity.com/paper/2401.06406/) (2024)
- [Sentiment Analysis of Movie Reviews Using BERT](https://mlsanity.com/paper/2502.18841/) (2025)
- [Auto-MLM: Improved Contrastive Learning for Self-supervised Multi-lingual Knowledge Retrieval](https://mlsanity.com/paper/2203.16187/) (2022)
- [KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model](https://mlsanity.com/paper/2311.11564/) (2023)

## Abstract (arXiv)

Knowledge of a disease includes information of various aspects of the disease, such as signs and symptoms, diagnosis and treatment. This disease knowledge is critical for many health-related and biomedical tasks, including consumer health question answering, medical language inference and disease name recognition. While pre-trained language models like BERT have shown success in capturing syntactic, semantic, and world knowledge from text, we find they can be further complemented by specific information like knowledge of symptoms, diagnoses, treatments, and other disease aspects. Hence, we integrate BERT with disease knowledge for improving these important tasks. Specifically, we propose a new disease knowledge infusion training procedure and evaluate it on a suite of BERT models including BERT, BioBERT, SciBERT, ClinicalBERT, BlueBERT, and ALBERT. Experiments over the three tasks show that these models can be enhanced in nearly all cases, demonstrating the viability of disease knowledge infusion. For example, accuracy of BioBERT on consumer health question answering is improved from 68.29% to 72.09%, while new SOTA results are observed in two datasets. We make our data and code freely available.