mlsanityData to 7 Oct 2026

Decomposed Prompt Tuning via Low-Rank Reparameterization

arXiv 2310.10094 ↗First posted 16 Oct 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary. In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low intrinsic rank characteristic. With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

36 of 51 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Layer Normalizationcited by 5,579
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
24 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2021GPT Understands, Toocited by 715
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  9. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  12. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  13. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  14. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  15. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  16. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  17. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  18. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  19. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  20. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  21. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  22. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  23. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
  24. 2023One Network, Many Masks: Towards More Parameter-Efficient Transfer Learningcited by 18

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 5.

8 papers.

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  2. 2025LatentEvolve: Self-Evolving Test-Time Scaling in Latent Spacecited by 8
  3. 2025PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuningcited by 6
  4. 2024Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classificationcited by 4
  5. 2024Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusioncited by 3
  6. 2025ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 3
  7. 2025TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removalcited by 1
  8. 2025CrossPT: Exploring Cross-Task Transferability through Multi-Task Prompt Tuningcited by 0