# Integrating Graph Contextualized Knowledge into Pre-trained Language Models

arXiv: https://arxiv.org/abs/1912.00147 · HTML: https://mlsanity.com/paper/1912.00147/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2019.

## Built on by (most cited first, 26 of 26)

- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study](https://mlsanity.com/paper/2308.14321/) (2023)
- [Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancement](https://mlsanity.com/paper/2412.13575/) (2024)
- [Combining pre-trained language models and structured knowledge](https://mlsanity.com/paper/2101.12294/) (2021)
- [Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation](https://mlsanity.com/paper/2206.01988/) (2022)
- [Multimodal Learning on Graphs for Disease Relation Extraction](https://mlsanity.com/paper/2203.08893/) (2022)
- [Large Knowledge Model: Perspectives and Challenges](https://mlsanity.com/paper/2312.02706/) (2023)
- [Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining](https://mlsanity.com/paper/2304.14204/) (2023)
- [Structure Inducing Pre-Training](https://mlsanity.com/paper/2103.10334/) (2021)
- [Integrating UMLS Knowledge into Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2310.02778/) (2023)
- [Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval](https://mlsanity.com/paper/2109.02789/) (2021)
- [ECOLA: Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations](https://mlsanity.com/paper/2203.09590/) (2022)
- [A Retrieve-and-Read Framework for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2212.09724/) (2022)
- [Coreference Resolution for the Biomedical Domain: A Survey](https://mlsanity.com/paper/2109.12424/) (2021)
- [Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models](https://mlsanity.com/paper/2312.03633/) (2023)
- [Focusing Knowledge-based Graph Argument Mining via Topic Modeling](https://mlsanity.com/paper/2102.02086/) (2021)
- [A Survey of Knowledge Enhanced Pre-trained Models](https://mlsanity.com/paper/2110.00269/) (2021)
- [Protein Representation Learning via Knowledge Enhanced Primary Structure Modeling](https://mlsanity.com/paper/2301.13154/) (2023)
- [K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering](https://mlsanity.com/paper/2109.10547/) (2021)
- [Adapter-TST: A Parameter Efficient Method for Multiple-Attribute Text Style Transfer](https://mlsanity.com/paper/2305.05945/) (2023)
- [Question Answering with Deep Neural Networks for Semi-Structured Heterogeneous Genealogical Knowledge Graphs](https://mlsanity.com/paper/2307.16214/) (2023)
- [BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment](https://mlsanity.com/paper/2509.07588/) (2025)
- [MLRIP: Pre-training a military language representation model with informative factual knowledge and professional knowledge base](https://mlsanity.com/paper/2207.13929/) (2022)
- [Make BERT-based Chinese Spelling Check Model Enhanced by Layerwise Attention and Gaussian Mixture Model](https://mlsanity.com/paper/2312.16623/) (2023)

## Abstract (arXiv)

Complex node interactions are common in knowledge graphs, and these interactions also contain rich knowledge information. However, traditional methods usually treat a triple as a training unit during the knowledge representation learning (KRL) procedure, neglecting contextualized information of the nodes in knowledge graphs (KGs). We generalize the modeling object to a very general form, which theoretically supports any subgraph extracted from the knowledge graph, and these subgraphs are fed into a novel transformer-based model to learn the knowledge embeddings. To broaden usage scenarios of knowledge, pre-trained language models are utilized to build a model that incorporates the learned knowledge representations. Experimental results demonstrate that our model achieves the state-of-the-art performance on several medical NLP tasks, and improvement above TransE indicates that our KRL method captures the graph contextualized information effectively.