mlsanityData to 7 Oct 2026

Prefix-Tuning: Optimizing Continuous Prompts for Generation

arXiv 2101.00190 ↗First posted 1 Jan 2021cs.CLcited by 3,878

Not on the map. Why?

Cited by 258 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Fine-tuning is the de facto way to leverage large pretrained language models to perform downstream tasks. However, it modifies all the language model parameters and therefore necessitates storing a full copy for each task. In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen, but optimizes a small continuous task-specific vector (called the prefix). Prefix-tuning draws inspiration from prompting, allowing subsequent tokens to attend to this prefix as if it were "virtual tokens". We apply prefix-tuning to GPT-2 for table-to-text generation and to BART for summarization. We find that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with topics unseen during training.

Key facts. Cited by 3,878 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

34 of 55 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
22 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  3. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  8. 2017Learning multiple visual domains with residual adapterscited by 553
  9. 2019Text Summarization with Pretrained Encoderscited by 553
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  12. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  13. 2020Extractive Summarization as Text Matchingcited by 189
  14. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  15. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  16. 2020Text-to-Text Pre-Training for Data-to-Text Taskscited by 104
  17. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  18. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  19. 2019Pragmatically Informative Text Generationcited by 32
  20. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  21. 2019Can Unconditional Language Models Recover Arbitrary Sentences?cited by 15
  22. 2020Conditioned Natural Language Generation using only Unconditioned Language Model: An Explorationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 133; 2022: 461; 2023: 911; 2024: 1010; 2025: 769; 2026: 594.

3,878 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  9. 2021Learning to Prompt for Continual Learningcited by 938
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  14. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  15. 2023Large Language Models as Optimizerscited by 830
  16. 2023Steering Language Models With Activation Engineeringcited by 824
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  18. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  19. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  20. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  21. 2021GPT Understands, Toocited by 715
  22. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  23. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  24. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  25. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  26. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  27. 2024A Survey on Large Language Models for Code Generationcited by 646
  28. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  29. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  30. 2023A Comprehensive Overview of Large Language Modelscited by 639
  31. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  32. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  33. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  34. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  35. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  36. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  37. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  38. 2023Segment Everything Everywhere All at Oncecited by 546

50 most cited of 3,878 All 3,878 (JSON)