mlsanityData to 7 Oct 2026

PPT: Pre-trained Prompt Tuning for Few-shot Learning

arXiv 2109.04332 ↗First posted 9 Sep 2021cs.CLcited by 232

Not on the map. Why?

Abstract (arXiv)

Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model fine-tuning when downstream data are sufficient, whereas it performs much worse under few-shot learning settings, which may hinder the application of prompt tuning in practice. We attribute this low performance to the manner of initializing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. We name this Pre-trained Prompt Tuning framework "PPT". To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and efficient for using large-scale PLMs in practice.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

38 of 52 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
26 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  7. 2021GPT Understands, Toocited by 715
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  9. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  10. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  11. 2021True Few-Shot Learning with Language Modelscited by 343
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  13. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  14. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  15. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  16. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  17. 2018A Simple Method for Commonsense Reasoningcited by 223
  18. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  19. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  20. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  21. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  22. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  23. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  24. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  25. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  26. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 74; 2023: 74; 2024: 43; 2025: 28; 2026: 5.

232 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  5. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  9. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  10. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  11. 2022Data Contamination: From Memorization to Exploitationcited by 234
  12. 2023Continual Pre-training of Language Modelscited by 194
38 more
  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  3. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  4. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  5. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  6. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  7. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  8. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  9. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  10. 2022Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learningcited by 85
  11. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  12. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  13. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  14. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  15. 2022Feature-Proxy Transformer for Few-Shot Segmentationcited by 58
  16. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  17. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  18. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  19. 2023Learning from models beyond fine-tuningcited by 47
  20. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  21. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  22. 2023VLCounter: Text-aware Visual Representation for Zero-Shot Object Countingcited by 45
  23. 2022Continual Prompt Tuning for Dialog State Trackingcited by 44
  24. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  25. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  26. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  27. 2023Pre-Training to Learn in Contextcited by 41
  28. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  29. 2024Data Selection via Optimal Control for Language Modelscited by 38
  30. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  31. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  32. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  33. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  34. 2023Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extractioncited by 32
  35. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  36. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  37. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  38. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29

50 most cited of 232 All 232 (JSON)