# Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

arXiv: https://arxiv.org/abs/2004.14224 · HTML: https://mlsanity.com/paper/2004.14224/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 26 of 26)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](https://mlsanity.com/paper/2201.08860/) (2022)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [Sequence-to-Sequence Knowledge Graph Completion and Question Answering](https://mlsanity.com/paper/2203.10321/) (2022)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Knowledge Graph Embedding: A Survey from the Perspective of Representation Spaces](https://mlsanity.com/paper/2211.03536/) (2022)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications](https://mlsanity.com/paper/2306.02592/) (2023)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Combining pre-trained language models and structured knowledge](https://mlsanity.com/paper/2101.12294/) (2021)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Structure Inducing Pre-Training](https://mlsanity.com/paper/2103.10334/) (2021)
- [Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks](https://mlsanity.com/paper/2410.01985/) (2024)
- [CuriousLLM: Elevating Multi-Document Question Answering with LLM-Enhanced Knowledge Graph Reasoning](https://mlsanity.com/paper/2404.09077/) (2024)
- [GraphWOZ: Dialogue Management with Conversational Knowledge Graphs](https://mlsanity.com/paper/2211.12852/) (2022)
- [Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them](https://mlsanity.com/paper/2503.22006/) (2025)
- [A Survey of Knowledge Enhanced Pre-trained Models](https://mlsanity.com/paper/2110.00269/) (2021)
- [Detecting Euphemisms with Literal Descriptions and Visual Imagery](https://mlsanity.com/paper/2211.04576/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Semantic Categorization of Social Knowledge for Commonsense Question Answering](https://mlsanity.com/paper/2109.05168/) (2021)
- [ConcEPT: Concept-Enhanced Pre-Training for Language Models](https://mlsanity.com/paper/2401.05669/) (2024)
- [Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems](https://mlsanity.com/paper/2107.00315/) (2021)

## Abstract (arXiv)

In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.