mlsanityData to 14 Oct 2026

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

arXiv 2303.02861 ↗First posted 6 Mar 2023cs.CLcited by 115

Not on the map. Why?

Abstract (arXiv)

Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task. Extensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-art methods, including the full finetuning baseline in some cases, despite only tuning 0.035% as many task-specific parameters.

Key facts. Cited by 115 papers in the map (data through 14 Oct 2026). Builds on 72 of its 92 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

72 of 92 references.

References · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  14. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  15. 2018Neural Network Acceptability Judgmentscited by 958
  16. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  17. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  18. 2017A Survey on Multi-Task Learningcited by 840
  19. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  20. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  21. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  22. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  23. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  24. 2018Born Again Neural Networkscited by 546
  25. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  26. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  27. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  28. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  29. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  30. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  31. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  32. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  33. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  34. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  35. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  36. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  37. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  38. 2021Training Neural Networks with Fixed Sparse Maskscited by 214

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 27; 2024: 41; 2025: 34; 2026: 13.

115 papers.

Sort:

By area:

  1. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  3. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  4. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  5. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  6. 2023Efficient Large Language Models: A Surveycited by 186
  7. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  8. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  9. 2023PromptIR: Prompting for All-in-One Blind Image Restorationcited by 114
  10. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Modelscited by 78
  11. 2023Learning from models beyond fine-tuningcited by 48
  12. 2023LLaFS: When Large Language Models Meet Few-Shot Segmentationcited by 48
38 more
  1. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  2. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 43
  3. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
  4. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  5. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  6. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  7. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  8. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  9. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  10. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  11. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  12. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  13. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  14. 2023Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modelingcited by 20
  15. 2025System Prompt Optimization with Meta-Learningcited by 18
  16. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  17. 2024Recovering the Pre-Fine-Tuning Weights of Generative Modelscited by 16
  18. 2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 16
  19. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
  20. 2024Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generationcited by 15
  21. 2024PromptRR: Diffusion Models as Prompt Generators for Single Image Reflection Removalcited by 14
  22. 2023Exploring the Relationship between LLM Hallucinations and Prompt Linguistic Nuances: Readability, Formality, and Concretenesscited by 13
  23. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  24. 2023Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Modelscited by 12
  25. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  26. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  27. 2024Fourier Prompt Tuning for Modality-Incomplete Scene Segmentationcited by 11
  28. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  29. 2024FACTOID: FACtual enTailment fOr hallucInation Detectioncited by 10
  30. 2024Towards Infinite-Long Prefix in Transformercited by 10
  31. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 10
  32. 2023EPVT: Environment-aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognitioncited by 9
  33. 2024Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learningcited by 9
  34. 2025MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learningcited by 9
  35. 2025Dynamic Prompt Fusion for Multi-Task and Cross-Domain Adaptation in LLMscited by 9
  36. 2023Decomposed Prompt Tuning via Low-Rank Reparameterizationcited by 8
  37. 2024Prompt-driven Latent Domain Generalization for Medical Image Classificationcited by 8
  38. 2023Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Modelscited by 7

50 most cited of 115 All 115 (JSON)