mlsanityData to 7 Oct 2026

Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT

arXiv 2109.04810 ↗First posted 10 Sep 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. In this paper, we proposed Mixture-of-Partitions (MoP), an infusion approach that can handle a very large knowledge graph (KG) by partitioning it into smaller sub-graphs and infusing their specific knowledge into various BERT models using lightweight adapters. To leverage the overall factual knowledge for a target task, these sub-graph adapters are further fine-tuned along with the underlying BERT through a mixture layer. We evaluate our MoP with three biomedical BERTs (SciBERT, BioBERT, PubmedBERT) on six downstream tasks (inc. NLI, QA, Classification), and the results show that our MoP consistently enhances the underlying BERTs in task performance, and achieves new SOTA performances on five evaluated datasets.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

31 of 41 references.

References · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
19 more
  1. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  2. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  3. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  4. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  5. 2020Autoregressive Entity Retrievalcited by 333
  6. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  7. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  8. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  9. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  10. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  11. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  12. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  13. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  14. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  15. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  16. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  17. 2020Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognitioncited by 35
  18. 2020Results of the seventh edition of the BioASQ Challengecited by 28
  19. 2021Overview of BioASQ 2020: The eighth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answeringcited by 13

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 8; 2023: 2; 2024: 3.

14 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  3. 2022Knowledge Graph Embedding: A Survey from the Perspective of Representation Spacescited by 41
  4. 2022KCD: Knowledge Walks and Textual Cues Enhanced Political Perspective Detection in News Mediacited by 20
  5. 2024InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integrationcited by 19
  6. 2022KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understandingcited by 9
  7. 2022Mixture of Experts for Biomedical Question Answeringcited by 6
  8. 2022Can Pretrained Language Models (Yet) Reason Deductively?cited by 5
  9. 2022Review of Natural Language Processing in Pharmacologycited by 3
  10. 2022The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injectioncited by 2
  11. 2023UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognitioncited by 1
  12. 2023Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphscited by 1
2 more
  1. 2021Common Sense Knowledge Learning for Open Vocabulary Neural Reasoning: A First View into Chronic Disease Literaturecited by 0
  2. 2024Adapter-based Approaches to Knowledge-enhanced Language Models -- A Surveycited by 0