mlsanityData to 7 Oct 2026

K-BERT: Enabling Language Representation with Knowledge Graph

arXiv 1909.07606 ↗First posted 17 Sep 2019cs.CL cs.LGcited by 303

Not on the map. Why?

Abstract (arXiv)

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by equipped with a KG without pre-training by-self because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 59; 2022: 79; 2023: 71; 2024: 52; 2025: 23; 2026: 19.

303 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  4. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  5. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  6. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  7. 2022A Review on Language Models as Knowledge Basescited by 156
  8. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  9. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  10. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 114
  11. 2021CAT: Cross Attention in Vision Transformercited by 92
  12. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
38 more
  1. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  4. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  5. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
  6. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  7. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  8. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  9. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  10. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  11. 2022RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Groundingcited by 47
  12. 2024Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrievalcited by 45
  13. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 44
  14. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  15. 2022A Decade of Knowledge Graphs in Natural Language Processing: A Surveycited by 40
  16. 2023ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Modelcited by 40
  17. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  18. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
  19. 2024Generative Artificial Intelligence: A Systematic Review and Applicationscited by 37
  20. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  21. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  22. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  23. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  24. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  25. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  26. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 30
  27. 2021RAGA: Relation-aware Graph Attention Networks for Global Entity Alignmentcited by 29
  28. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  29. 2023Logical Message Passing Networks with One-hop Inference on Atomic Formulascited by 28
  30. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 28
  31. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  32. 2024Knowledge Graph-Enhanced Large Language Models via Path Selectioncited by 28
  33. 2023InstructProtein: Aligning Human and Protein Language via Knowledge Instructioncited by 27
  34. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  35. 2021Combining pre-trained language models and structured knowledgecited by 25
  36. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  37. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  38. 2021SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Miningcited by 24

50 most cited of 303 All 303 (JSON)