mlsanityData to 7 Oct 2026

AdaPrompt: Adaptive Model Training for Prompt-based NLP

arXiv 2202.04824 ↗First posted 10 Feb 2022cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Prompt-based learning, with its capability to tackle zero-shot and few-shot NLP tasks, has gained much attention in community. The main idea is to bridge the gap between NLP downstream tasks and language modeling (LM), by mapping these tasks into natural language prompts, which are then filled by pre-trained language models (PLMs). However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining. First, prompt information is not necessarily sufficiently present during LM pretraining. Second, task-specific data are not necessarily well represented during pretraining. We address these two issues by proposing AdaPrompt, adaptively retrieving external data for continual pretraining of PLMs by making use of both task and prompt characteristics. In addition, we make use of knowledge in Natural Language Inference models for deriving adaptive verbalizers. Experimental results on five NLP benchmarks show that AdaPrompt can improve over standard PLMs in few-shot settings. In addition, in zero-shot settings, our method outperforms standard prompt-based methods by up to 26.35% relative error reduction.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

40 of 55 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
28 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  9. 2019Defending Against Neural Fake Newscited by 553
  10. 2021True Few-Shot Learning with Language Modelscited by 343
  11. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  12. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  15. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  16. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  17. 2021Template-Based Named Entity Recognition Using BARTcited by 153
  18. 2020CTRLsum: Towards Generic Controllable Text Summarizationcited by 98
  19. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  20. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  21. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  22. 2019An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Modelscited by 39
  23. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  24. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  25. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  26. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  27. 2019Using Similarity Measures to Select Pretraining Data for NERcited by 15
  28. 2017Data Selection Strategies for Multi-Domain Sentiment Analysiscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 15; 2024: 4; 2025: 0; 2026: 1.

30 papers.

Sort:

By area:

  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  2. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  3. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  4. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  5. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  6. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  7. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
  8. 2022On the Compositional Generalization Gap of In-Context Learningcited by 24
  9. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  10. 2022Transformers are Adaptable Task Plannerscited by 15
  11. 2022Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognitioncited by 14
  12. 2022Retrieval Augmentation for Commonsense Reasoning: A Unified Approachcited by 14
18 more
  1. 2022MACSum: Controllable Summarization with Mixed Attributescited by 13
  2. 2022Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representationscited by 11
  3. 2023Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detectioncited by 10
  4. 2024Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schemacited by 10
  5. 2023Global Constraints with Prompting for Zero-Shot Event Argument Classificationcited by 8
  6. 2023Scaling In-Context Demonstrations with Structured Attentioncited by 8
  7. 2023ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervisioncited by 7
  8. 2023Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understandingcited by 4
  9. 2024SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topicscited by 4
  10. 2023A Communication Theory Perspective on Prompting Engineering Methods for Large Language Modelscited by 3
  11. 2024Prompt-tuning for Clickbait Detection via Text Summarizationcited by 3
  12. 2022Cross-domain Generalization for AMR Parsingcited by 2
  13. 2023Knowledge Rumination for Pre-trained Language Modelscited by 2
  14. 2023Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysiscited by 2
  15. 2023Constituency Parsing using LLMscited by 2
  16. 2024Cross-lingual Offensive Language Detection: A Systematic Review of Datasets, Transfer Approaches and Challengescited by 2
  17. 2026Reducing Hallucinations in LLM-based Scientific Literature Analysis Using Peer Context Outlier Detectioncited by 1
  18. 2023LaViP:Language-Grounded Visual Promptscited by 0