mlsanityData to 14 Oct 2026

On Transferability of Prompt Tuning for Natural Language Processing

arXiv 2111.06719 ↗First posted 12 Nov 2021cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-trained language models (PLMs), which can achieve comparable performance to full-parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intuitively, knowledge transfer can help to improve the efficiency. To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks on the same PLM and also to other PLMs with a cross-model projector trained on similar tasks; (2) when used as initialization, trained soft prompts of similar tasks and projected prompts of other PLMs can significantly accelerate training and also improve the performance of PT. Moreover, to explore what decides prompt transferability, we investigate various transferability indicators and find that the overlapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts stimulate PLMs is essential. Our findings show that prompt transfer is promising for improving PT, and further research shall focus more on prompts' stimulation to PLMs. The source code can be obtained from https://github.com/thunlp/Prompt-Transferability.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 12 Nov 2021.

Builds on

49 of 64 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2016Layer Normalizationcited by 5,669
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
37 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2019How Can We Know What Language Models Know?cited by 959
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  11. 2020Aligning AI With Shared Human Valuescited by 723
  12. 2021GPT Understands, Toocited by 716
  13. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  14. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  15. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  16. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  17. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  18. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  19. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  20. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  21. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  22. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  23. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  24. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  25. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  26. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  27. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  28. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  29. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  30. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  31. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  32. 2020Exploring and Predicting Transferability across NLP Taskscited by 128
  33. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  34. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  35. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  36. 2019Natural Language Understanding with the Quora Question Pairs Datasetcited by 51
  37. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 12; 2023: 33; 2024: 18; 2025: 10; 2026: 2.

75 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  3. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  4. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  5. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80
  6. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  7. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  8. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  9. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  10. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  11. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
  12. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
38 more
  1. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  2. 2023Dual-path Adaptation from Image to Video Transformerscited by 41
  3. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  4. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  5. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  6. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 30
  7. 2023Continual Learning with Low Rank Adaptationcited by 29
  8. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  9. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28
  10. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  11. 2023How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Promptscited by 23
  12. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compressioncited by 22
  13. 2023Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modelingcited by 20
  14. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  15. 2023Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Educationcited by 18
  16. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  17. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  18. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  19. 2022Prompt Tuning with Soft Context Sharing for Vision-Language Modelscited by 13
  20. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  21. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  22. 2023Dynamic Prompting: A Unified Framework for Prompt Tuningcited by 12
  23. 2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
  24. 2023One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NERcited by 11
  25. 2023TaskWeb: Selecting Better Source Tasks for Multi-task NLPcited by 11
  26. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  27. 2023PersonaPKT: Building Personalized Dialogue Agents via Parameter-efficient Knowledge Transfercited by 9
  28. 2023Can discrete information extraction prompts generalize across language models?cited by 8
  29. 2024UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Causecited by 8
  30. 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
  31. 2023Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Modelscited by 7
  32. 2025What Makes a Good Natural Language Prompt?cited by 7
  33. 2022Extensible Prompts for Language Models on Zero-shot Language Style Customizationcited by 6
  34. 2023Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inferencecited by 6
  35. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  36. 2025Understanding Prompt Tuning and In-Context Learning via Meta-Learningcited by 6
  37. 2023Black-box Prompt Tuning with Subspace Learningcited by 5
  38. 2023PromptSum: Parameter-Efficient Controllable Abstractive Summarizationcited by 4

50 most cited of 75 All 75 (JSON)