# KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning

arXiv: https://arxiv.org/abs/2012.03551 · HTML: https://mlsanity.com/paper/2012.03551/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Structure Inducing Pre-Training](https://mlsanity.com/paper/2103.10334/) (2021)
- [Bridging the Language Gap: Knowledge Injected Multilingual Question Answering](https://mlsanity.com/paper/2304.03159/) (2023)
- [ArK: Augmented Reality with Knowledge Interactive Emergent Ability](https://mlsanity.com/paper/2305.00970/) (2023)
- [Commonsense Knowledge Transfer for Pre-trained Language Models](https://mlsanity.com/paper/2306.02388/) (2023)
- [A Domain-Agnostic Neurosymbolic Approach for Big Social Data Analysis: Evaluating Mental Health Sentiment on Social Media during COVID-19](https://mlsanity.com/paper/2411.07163/) (2024)
- [Pre-training Language Models with Deterministic Factual Knowledge](https://mlsanity.com/paper/2210.11165/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [ChatGPT Alternative Solutions: Large Language Models Survey](https://mlsanity.com/paper/2403.14469/) (2024)

## Abstract (arXiv)

Recent studies on pre-trained language models have demonstrated their ability to capture factual knowledge and applications in knowledge-aware downstream tasks. In this work, we present a language model pre-training framework guided by factual knowledge completion and verification, and use the generative and discriminative approaches cooperatively to learn the model. Particularly, we investigate two learning schemes, named two-tower scheme and pipeline scheme, in training the generator and discriminator with shared parameter. Experimental results on LAMA, a set of zero-shot cloze-style question answering tasks, show that our model contains richer factual knowledge than the conventional pre-trained language models. Furthermore, when fine-tuned and evaluated on the MRQA shared tasks which consists of several machine reading comprehension datasets, our model achieves the state-of-the-art performance, and gains large improvements on NewsQA (+1.26 F1) and TriviaQA (+1.56 F1) over RoBERTa.