# AdaPrompt: Adaptive Model Training for Prompt-based NLP

arXiv: https://arxiv.org/abs/2202.04824 · HTML: https://mlsanity.com/paper/2202.04824/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [CTRLsum: Towards Generic Controllable Text Summarization](https://mlsanity.com/paper/2012.04281/) (2020)

## Built on by (most cited first, 30 of 30)

- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [MolXPT: Wrapping Molecules with Text for Generative Pre-training](https://mlsanity.com/paper/2305.10688/) (2023)
- [CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition](https://mlsanity.com/paper/2303.11313/) (2023)
- [GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective](https://mlsanity.com/paper/2211.08073/) (2022)
- [DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer](https://mlsanity.com/paper/2312.03724/) (2023)
- [On the Compositional Generalization Gap of In-Context Learning](https://mlsanity.com/paper/2211.08473/) (2022)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Transformers are Adaptable Task Planners](https://mlsanity.com/paper/2207.02442/) (2022)
- [Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2210.07032/) (2022)
- [Retrieval Augmentation for Commonsense Reasoning: A Unified Approach](https://mlsanity.com/paper/2210.12887/) (2022)
- [MACSum: Controllable Summarization with Mixed Attributes](https://mlsanity.com/paper/2211.05041/) (2022)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)
- [Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection](https://mlsanity.com/paper/2311.01025/) (2023)
- [Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema](https://mlsanity.com/paper/2412.20942/) (2024)
- [Global Constraints with Prompting for Zero-Shot Event Argument Classification](https://mlsanity.com/paper/2302.04459/) (2023)
- [Scaling In-Context Demonstrations with Structured Attention](https://mlsanity.com/paper/2307.02690/) (2023)
- [ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervision](https://mlsanity.com/paper/2307.01448/) (2023)
- [Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding](https://mlsanity.com/paper/2310.11721/) (2023)
- [SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics](https://mlsanity.com/paper/2410.01946/) (2024)
- [A Communication Theory Perspective on Prompting Engineering Methods for Large Language Models](https://mlsanity.com/paper/2310.18358/) (2023)
- [Prompt-tuning for Clickbait Detection via Text Summarization](https://mlsanity.com/paper/2404.11206/) (2024)
- [Cross-domain Generalization for AMR Parsing](https://mlsanity.com/paper/2210.12445/) (2022)
- [Knowledge Rumination for Pre-trained Language Models](https://mlsanity.com/paper/2305.08732/) (2023)
- [Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis](https://mlsanity.com/paper/2306.07664/) (2023)
- [Constituency Parsing using LLMs](https://mlsanity.com/paper/2310.19462/) (2023)
- [Cross-lingual Offensive Language Detection: A Systematic Review of Datasets, Transfer Approaches and Challenges](https://mlsanity.com/paper/2401.09244/) (2024)
- [Reducing Hallucinations in LLM-based Scientific Literature Analysis Using Peer Context Outlier Detection](https://mlsanity.com/paper/2604.01461/) (2026)
- [LaViP:Language-Grounded Visual Prompts](https://mlsanity.com/paper/2312.10945/) (2023)

## Abstract (arXiv)

Prompt-based learning, with its capability to tackle zero-shot and few-shot NLP tasks, has gained much attention in community. The main idea is to bridge the gap between NLP downstream tasks and language modeling (LM), by mapping these tasks into natural language prompts, which are then filled by pre-trained language models (PLMs). However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining. First, prompt information is not necessarily sufficiently present during LM pretraining. Second, task-specific data are not necessarily well represented during pretraining. We address these two issues by proposing AdaPrompt, adaptively retrieving external data for continual pretraining of PLMs by making use of both task and prompt characteristics. In addition, we make use of knowledge in Natural Language Inference models for deriving adaptive verbalizers. Experimental results on five NLP benchmarks show that AdaPrompt can improve over standard PLMs in few-shot settings. In addition, in zero-shot settings, our method outperforms standard prompt-based methods by up to 26.35% relative error reduction.