mlsanityData to 7 Oct 2026

The Power of Scale for Parameter-Efficient Prompt Tuning

arXiv 2104.08691 ↗First posted 18 Apr 2021cs.CLcited by 3,769

Not on the map. Why?

Cited by 284 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's "few-shot" learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant in that large models are costly to share and serve, and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021), and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.

Key facts. Cited by 3,769 papers in the map (data through 7 Oct 2026). Builds on 33 of its 60 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

33 of 60 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
21 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  11. 2017Learning multiple visual domains with residual adapterscited by 553
  12. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  13. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  14. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  15. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  16. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  17. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  18. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  19. 2018DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehensioncited by 70
  20. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  21. 2020Few-shot Sequence Learning with Transformerscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 88; 2022: 407; 2023: 930; 2024: 1010; 2025: 780; 2026: 554.

3,769 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2022Visual Prompt Tuningcited by 1,674
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  13. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  15. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  16. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  17. 2021Learning to Prompt for Continual Learningcited by 938
  18. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  19. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  20. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  21. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  22. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  23. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  24. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  25. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  26. 2023Large Language Models as Optimizerscited by 830
  27. 2023Steering Language Models With Activation Engineeringcited by 824
  28. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  29. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  30. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  31. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  32. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  33. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  34. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  35. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  36. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  37. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  38. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679

50 most cited of 3,769 All 3,769 (JSON)