# Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

arXiv: https://arxiv.org/abs/2010.12688 · HTML: https://mlsanity.com/paper/2010.12688/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Open Domain Question Answering with A Unified Knowledge Interface](https://mlsanity.com/paper/2110.08417/) (2021)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [SKILL: Structured Knowledge Infusion for Large Language Models](https://mlsanity.com/paper/2205.08184/) (2022)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Database Reasoning Over Text](https://mlsanity.com/paper/2106.01074/) (2021)
- [KILM: Knowledge Injection into Encoder-Decoder Language Models](https://mlsanity.com/paper/2302.09170/) (2023)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Cognitive Semantic Communication Systems Driven by Knowledge Graph](https://mlsanity.com/paper/2202.11958/) (2022)
- [OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models](https://mlsanity.com/paper/2410.23703/) (2024)
- [LM-CORE: Language Models with Contextually Relevant External Knowledge](https://mlsanity.com/paper/2208.06458/) (2022)
- [Open Relation Modeling: Learning to Define Relations between Entities](https://mlsanity.com/paper/2108.09241/) (2021)
- [Knowledge-Augmented Language Models for Cause-Effect Relation Classification](https://mlsanity.com/paper/2112.08615/) (2021)
- [Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies](https://mlsanity.com/paper/2204.05879/) (2022)
- [DEER: Descriptive Knowledge Graph for Explaining Entity Relationships](https://mlsanity.com/paper/2205.10479/) (2022)
- [RoMe: A Robust Metric for Evaluating Natural Language Generation](https://mlsanity.com/paper/2203.09183/) (2022)
- [Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generation](https://mlsanity.com/paper/2311.09467/) (2023)
- [Decouple knowledge from parameters for plug-and-play language modeling](https://mlsanity.com/paper/2305.11564/) (2023)
- [Roof-Transformer: Divided and Joined Understanding with Knowledge Enhancement](https://mlsanity.com/paper/2112.06736/) (2021)
- [DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection](https://mlsanity.com/paper/2203.09598/) (2022)
- [ComDensE : Combined Dense Embedding of Relation-aware and Common Features for Knowledge Graph Completion](https://mlsanity.com/paper/2206.14925/) (2022)
- [Identifying and Aligning Medical Claims Made on Social Media with Medical Evidence](https://mlsanity.com/paper/2405.11219/) (2024)

## Abstract (arXiv)

Prior work on Data-To-Text Generation, the task of converting knowledge graph (KG) triples into natural text, focused on domain-specific benchmark datasets. In this paper, however, we verbalize the entire English Wikidata KG, and discuss the unique challenges associated with a broad, open-domain, large-scale verbalization. We further show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora. In contrast to the many architectures that have been developed to integrate these two sources, our approach converts the KG into natural text, allowing it to be seamlessly integrated into existing language models. It carries the further advantages of improved factual accuracy and reduced toxicity in the resulting language model. We evaluate this approach by augmenting the retrieval corpus in a retrieval language model and showing significant improvements on the knowledge intensive tasks of open domain QA and the LAMA knowledge probe.