# Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

arXiv: https://arxiv.org/abs/2108.02035 · HTML: https://mlsanity.com/paper/2108.02035/

## Key facts

- Cited by 183 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)

## Built on by (most cited first, 30 of 183)

- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021)
- [MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators](https://mlsanity.com/paper/2306.10900/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://mlsanity.com/paper/2306.04136/) (2023)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning](https://mlsanity.com/paper/2206.09363/) (2022)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks](https://mlsanity.com/paper/2206.08514/) (2022)
- [Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks](https://mlsanity.com/paper/2310.10830/) (2023)
- [Prompt-Learning for Fine-Grained Entity Typing](https://mlsanity.com/paper/2108.10604/) (2021)
- [MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding](https://mlsanity.com/paper/2410.21747/) (2024)
- [ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation](https://mlsanity.com/paper/2306.04344/) (2023)
- [Prototypical Verbalizer for Prompt-based Few-shot Tuning](https://mlsanity.com/paper/2203.09770/) (2022)
- [UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All](https://mlsanity.com/paper/2403.12532/) (2024)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [Web-Shepherd: Advancing PRMs for Reinforcing Web Agents](https://mlsanity.com/paper/2505.15277/) (2025)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Gradient-based Parameter Selection for Efficient Fine-Tuning](https://mlsanity.com/paper/2312.10136/) (2023)
- [Paradigm Shift in Natural Language Processing](https://mlsanity.com/paper/2109.12575/) (2021)
- [Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning](https://mlsanity.com/paper/2205.14704/) (2022)

## Abstract (arXiv)

Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.