mlsanityData to 7 Oct 2026

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

arXiv 2110.07602 ↗First posted 14 Oct 2021cs.CLcited by 686

Not on the map. Why?

Cited by 50 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning citeli2021prefix,qin2021learning optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.Our code and data are released at https://github.com/THUDM/P-tuning-v2.

Key facts. Cited by 686 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

35 of 57 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
23 more
  1. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  2. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  3. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  4. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  5. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  6. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  7. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  8. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  9. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  10. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  11. 2021Aspect Sentiment Quad Prediction as Paraphrase Generationcited by 99
  12. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  13. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  14. 2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
  15. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  16. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  17. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  18. 2021NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Predictioncited by 23
  19. 2021SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment Analysiscited by 23
  20. 2021Better Feature Integration for Named Entity Recognitioncited by 22
  21. 2021FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understandingcited by 17
  22. 2021Language Models are Good Translatorscited by 16
  23. 2021TopicRefine: Joint Topic Prediction and Dialogue Response Generation for Multi-turn End-to-End Dialogue Systemcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 103; 2023: 187; 2024: 205; 2025: 139; 2026: 47.

686 papers.

Sort:

By area:

  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  6. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  7. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  10. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  12. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
38 more
  1. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  4. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  5. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  6. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  7. 2023Personalize Segment Anything Model with One Shotcited by 250
  8. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  9. 2022Is GPT-3 a Good Data Annotator?cited by 208
  10. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  11. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  12. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  13. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  14. 2023Efficient Large Language Models: A Surveycited by 183
  15. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  16. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  17. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 164
  18. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  19. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  20. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  21. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  22. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  23. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  24. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  25. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  26. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  27. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  28. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  29. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  30. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  31. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  32. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  33. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  34. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  35. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  36. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  37. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  38. 2022Universal Prompt Tuning for Graph Neural Networkscited by 100

50 most cited of 686 All 686 (JSON)