mlsanityData to 14 Oct 2026

SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

arXiv 2110.07904 ↗First posted 15 Oct 2021cs.CLcited by 235

Not on the map. Why?

Abstract (arXiv)

There has been growing interest in parameter-efficient methods to apply pre-trained language models to downstream tasks. Building on the Prompt Tuning approach of Lester et al. (2021), which learns task-specific soft prompts to condition a frozen pre-trained model to perform different tasks, we propose a novel prompt-based transfer learning approach called SPoT: Soft Prompt Transfer. SPoT first learns a prompt on one or more source tasks and then uses it to initialize the prompt for a target task. We show that SPoT significantly boosts the performance of Prompt Tuning across many tasks. More remarkably, across all model sizes, SPoT matches or outperforms standard Model Tuning (which fine-tunes all model parameters) on the SuperGLUE benchmark, while using up to 27,000x fewer task-specific parameters. To understand where SPoT is most effective, we conduct a large-scale study on task transferability with 26 NLP tasks in 160 combinations, and demonstrate that many tasks can benefit each other via prompt transfer. Finally, we propose an efficient retrieval approach that interprets task prompts as task embeddings to identify similar tasks and predict the most transferable source tasks for a novel target task.

Key facts. Cited by 235 papers in the map (data through 14 Oct 2026). Builds on 84 of its 112 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

84 of 112 references.

References · 84

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  13. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  14. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  15. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  16. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  17. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  18. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  19. 2015Teaching Machines to Read and Comprehendcited by 1,049
  20. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  21. 2019How Can We Know What Language Models Know?cited by 959
  22. 2018Neural Network Acceptability Judgmentscited by 958
  23. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  24. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  25. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  26. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  27. 2021GPT Understands, Toocited by 716
  28. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  29. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  30. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  31. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  32. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  33. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  34. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  35. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  36. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  37. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  38. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 57; 2023: 76; 2024: 49; 2025: 33; 2026: 17.

235 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  5. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  7. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  8. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  9. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  10. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  11. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  12. 2022Active Example Selection for In-Context Learningcited by 196
38 more
  1. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  2. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  3. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  4. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  5. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  6. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  7. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 95
  8. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  9. 2023Modular Deep Learningcited by 83
  10. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  11. 2023MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRAcited by 80
  12. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  13. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  14. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  15. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  16. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 67
  17. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  18. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  19. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  20. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  21. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  22. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  23. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  24. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  25. 2021A Survey on Green Deep Learningcited by 52
  26. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  27. 2022Multitask Vision-Language Prompt Tuningcited by 51
  28. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  29. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  30. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
  31. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  32. 2022Continual Prompt Tuning for Dialog State Trackingcited by 44
  33. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  34. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 43
  35. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  36. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  37. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  38. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40

50 most cited of 235 All 235 (JSON)