mlsanityData to 14 Oct 2026

Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization

arXiv 2305.03937 ↗First posted 6 May 2023cs.CL cs.AIcited by 64

Not on the map. Why?

Abstract (arXiv)

Prompt tuning is one of the successful approaches for parameter-efficient tuning of pre-trained language models. Despite being arguably the most parameter-efficient (tuned soft prompts constitute <0.1% of total parameters), it typically performs worse than other efficient tuning methods and is quite sensitive to hyper-parameters. In this work, we introduce Residual Prompt Tuning - a simple and efficient method that significantly improves the performance and stability of prompt tuning. We propose to reparameterize soft prompt embeddings using a shallow network with a residual connection. Our experiments show that Residual Prompt Tuning significantly outperforms prompt tuning on SuperGLUE benchmark. Notably, our method reaches +7 points improvement over prompt tuning with T5-Base and allows to reduce the prompt length by 10x without hurting performance. In addition, we show that our approach is robust to the choice of learning rate and prompt initialization, and is effective in few-shot settings.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 6 May 2023.

Builds on

29 of 41 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2016Layer Normalizationcited by 5,669
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
17 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  9. 2021GPT Understands, Toocited by 716
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  12. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  13. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  14. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  15. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  16. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  17. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 20; 2025: 31; 2026: 6.

64 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  2. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  3. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  4. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  5. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 30
  6. 2023Prompting Language-Informed Distribution for Compositional Zero-Shot Learningcited by 24
  7. 2023Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modelingcited by 20
  8. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  9. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  10. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  11. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  12. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 14
38 more
  1. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  2. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  3. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
  4. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  5. 2025EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inferencecited by 9
  6. 2025Advancing Prompt-Based Methods for Replay-Independent General Continual Learningcited by 9
  7. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  8. 2023Decomposed Prompt Tuning via Low-Rank Reparameterizationcited by 8
  9. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  10. 2024Soft Prompt Tuning for Cross-Lingual Transfer: When Less is Morecited by 7
  11. 2025A Survey on Prompt Tuningcited by 7
  12. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  13. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  14. 2025PaCA: Partial Connection Adaptation for Efficient Fine-Tuningcited by 6
  15. 2025PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuningcited by 6
  16. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 6
  17. 2024Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningcited by 5
  18. 2024The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Modelscited by 5
  19. 2025Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformationscited by 5
  20. 2025Weight Spectra Induced Efficient Model Adaptationcited by 5
  21. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  22. 2025MEPT: Mixture of Expert Prompt Tuning as a Manifold Mappercited by 5
  23. 2025Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptationcited by 5
  24. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 5
  25. 2024LayerNorm: A key component in parameter-efficient fine-tuningcited by 4
  26. 2024Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuningcited by 4
  27. 2024BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptationcited by 4
  28. 2025DenseLoRA: Dense Low-Rank Adaptation of Large Language Modelscited by 4
  29. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  30. 2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
  31. 2025LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysiscited by 4
  32. 2024StablePT: Towards Stable Prompting for Few-shot Learning via Input Separationcited by 3
  33. 2024Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusioncited by 3
  34. 2025ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 3
  35. 2025Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Productcited by 3
  36. 2025Restoring Pruned Large Language Models via Lost Component Compensationcited by 3
  37. 2026SoftSkill: Behavioral Compression for Contextual Adaptationcited by 3
  38. 2024SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddingscited by 2

50 most cited of 64 All 64 (JSON)