mlsanityData to 7 Oct 2026

Knowledge Prompts: Injecting World Knowledge into Language Models through Soft Prompts

arXiv 2210.04726 ↗First posted 10 Oct 2022cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Soft prompts have been recently proposed as a tool for adapting large frozen language models (LMs) to new tasks. In this work, we repurpose soft prompts to the task of injecting world knowledge into LMs. We introduce a method to train soft prompts via self-supervised learning on data from knowledge bases. The resulting soft knowledge prompts (KPs) are task independent and work as an external memory of the LMs. We perform qualitative and quantitative experiments and demonstrate that: (1) KPs can effectively model the structure of the training data; (2) KPs can be used to improve the performance of LMs in different knowledge intensive tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

18 of 31 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  9. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  10. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  11. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  12. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
6 more
  1. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  2. 2015Large-scale Simple Question Answering with Memory Networkscited by 127
  3. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  4. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  5. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  6. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2.

6 papers.

  1. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  2. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  3. 2023Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Modelscited by 36
  4. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  5. 2023Memory Augmented Language Models through Mixture of Word Expertscited by 9
  6. 2024Infusing Knowledge into Large Language Models with Contextual Promptscited by 1