# Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study

arXiv: https://arxiv.org/abs/2106.09700 · HTML: https://mlsanity.com/paper/2106.09700/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Modeling polypharmacy side effects with graph convolutional networks](https://mlsanity.com/paper/1802.00543/) (2018)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Inductive Entity Representations from Text via Link Prediction](https://mlsanity.com/paper/2010.03496/) (2020)
- [A Review of Biomedical Datasets Relating to Drug Discovery: A Knowledge Graph Perspective](https://mlsanity.com/paper/2102.10062/) (2021)
- [Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings](https://mlsanity.com/paper/2006.13774/) (2020)
- [Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery](https://mlsanity.com/paper/2105.10488/) (2021)
- [DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections](https://mlsanity.com/paper/2102.13247/) (2021)

## Built on by (most cited first, 23 of 23)

- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [Can Large Language Models Unlock Novel Scientific Research Ideas?](https://mlsanity.com/paper/2409.06185/) (2024)
- [A Computational Inflection for Scientific Discovery](https://mlsanity.com/paper/2205.02007/) (2022)
- [Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders](https://mlsanity.com/paper/2405.17044/) (2024)
- [Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2110.08173/) (2021)
- [IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery](https://mlsanity.com/paper/2504.16728/) (2025)
- [DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articles](https://mlsanity.com/paper/2207.01079/) (2022)
- [I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning](https://mlsanity.com/paper/2208.09828/) (2022)
- [Bursting Scientific Filter Bubbles: Boosting Innovation via Novel Author Discovery](https://mlsanity.com/paper/2108.05669/) (2021)
- [SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design](https://mlsanity.com/paper/2307.11694/) (2023)
- [Rethinking Language Models as Symbolic Knowledge Graphs](https://mlsanity.com/paper/2308.13676/) (2023)
- [HypER: Literature-grounded Hypothesis Generation and Distillation with Provenance](https://mlsanity.com/paper/2506.12937/) (2025)
- [NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation](https://mlsanity.com/paper/2312.05979/) (2023)
- [CascadER: Cross-Modal Cascading for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2205.08012/) (2022)
- [Knowledge Graph Error Detection with Contrastive Confidence Adaption](https://mlsanity.com/paper/2312.12108/) (2023)
- [BLIAM: Literature-based Data Synthesis for Synergistic Drug Combination Prediction](https://mlsanity.com/paper/2302.06860/) (2023)
- [Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Research](https://mlsanity.com/paper/2312.03497/) (2023)
- [Flow-Modulated Scoring for Semantic-Aware Knowledge Graph Completion](https://mlsanity.com/paper/2506.23137/) (2025)
- [MindFlow: Mind Supernet Powered Thinking Flows for Research Idea Innovation](https://mlsanity.com/paper/2610.11966/) (2026)
- [Pseudo-Riemannian Embedding Models for Multi-Relational Graph Representations](https://mlsanity.com/paper/2212.03720/) (2022)
- [What's In Your Field? Mapping Scientific Research with Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2503.09894/) (2025)
- [Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research](https://mlsanity.com/paper/2505.06964/) (2025)

## Abstract (arXiv)

Biomedical knowledge graphs (KGs) hold rich information on entities such as diseases, drugs, and genes. Predicting missing links in these graphs can boost many important applications, such as drug design and repurposing. Recent work has shown that general-domain language models (LMs) can serve as "soft" KGs, and that they can be fine-tuned for the task of KG completion. In this work, we study scientific LMs for KG completion, exploring whether we can tap into their latent knowledge to enhance biomedical link prediction. We evaluate several domain-specific LMs, fine-tuning them on datasets centered on drugs and diseases that we represent as KGs and enrich with textual entity descriptions. We integrate the LM-based models with KG embedding models, using a router method that learns to assign each input example to either type of model and provides a substantial boost in performance. Finally, we demonstrate the advantage of LM models in the inductive setting with novel scientific entities. Our datasets and code are made publicly available.