# Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding arXiv: https://arxiv.org/abs/2210.08536 ยท HTML: https://mlsanity.com/paper/2210.08536/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 40 of its 57 references that are in the map. - First posted on arXiv 16 Oct 2022. ## Builds on (most cited first, 30 of 40) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020) - [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021) - [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019) - [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015) - [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020) - [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019) - [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021) - [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021) - [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019) - [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019) - [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019) - [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020) - [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019) - [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019) - [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019) - [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018) - [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021) - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021) - [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021) - [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018) - [KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning](https://mlsanity.com/paper/2009.12677/) (2020) ## Built on by (most cited first, 9 of 9) - [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023) - [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022) - [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023) - [Large Language Models Can Better Understand Knowledge Graphs Than We Thought](https://mlsanity.com/paper/2402.11541/) (2024) - [Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2501.14300/) (2025) - [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024) - [Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning](https://mlsanity.com/paper/2408.17070/) (2024) - [HugNLP: A Unified and Comprehensive Library for Natural Language Processing](https://mlsanity.com/paper/2302.14286/) (2023) - [MiCU: End-to-End Smart Home Command Understanding with Large Language Model](https://mlsanity.com/paper/2606.01099/) (2026) ## Abstract (arXiv) Knowledge-enhanced Pre-trained Language Model (PLM) has recently received significant attention, which aims to incorporate factual knowledge into PLMs. However, most existing methods modify the internal structures of fixed types of PLMs by stacking complicated modules, and introduce redundant and irrelevant factual knowledge from knowledge bases (KBs). In this paper, to address these problems, we introduce a seminal knowledge prompting paradigm and further propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be flexibly combined with existing mainstream PLMs. Specifically, we first construct a knowledge sub-graph from KBs for each context. Then we design multiple continuous prompts rules and transform the knowledge sub-graph into natural language prompts. To further leverage the factual knowledge from these prompts, we propose two novel knowledge-aware self-supervised tasks including prompt relevance inspection and masked prompt modeling. Extensive experiments on multiple natural language understanding (NLU) tasks show the superiority of KP-PLM over other state-of-the-art methods in both full-resource and low-resource settings.