# Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product

arXiv: https://arxiv.org/abs/2502.12200 · HTML: https://mlsanity.com/paper/2502.12200/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2206.06522/) (2022)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://mlsanity.com/paper/1810.12885/) (2018)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [LoRA-GA: Low-Rank Adaptation with Gradient Approximation](https://mlsanity.com/paper/2407.05000/) (2024)
- [Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2303.02861/) (2023)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)

## Built on by (most cited first, 3 of 3)

- [All You Need is One: Capsule Prompt Tuning with a Single Vector](https://mlsanity.com/paper/2510.16670/) (2025)
- [PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark](https://mlsanity.com/paper/2511.21285/) (2025)
- [Cognivia: A Cognitive Behavioral Therapy Copilot for Evidence-Based Mental Healthcare](https://mlsanity.com/paper/2607.25681/) (2026)

## Abstract (arXiv)

Prompt tuning (PT) offers a cost-effective alternative to fine-tuning large-scale pre-trained language models (PLMs), requiring only a few parameters in soft prompt tokens added before the input text. However, existing PT approaches face two significant issues: (i) They overlook intrinsic semantic associations between soft prompt tokens, leading to high discreteness and limited interactions, thus reducing the model's comprehension and effectiveness in complex tasks. (ii) Due to the complexity of downstream tasks, long soft prompt is necessitated to improve performance, but prompt length correlates positively with memory usage and computational costs. Achieving high efficiency and performance remains an ongoing challenge. To address these issues, we propose a novel Low-parameters prompt tuning (LAMP) method, which leverages prompt decomposition and compressed outer product. Specifically, the prompt decomposition module employs Truncated SVD to reduce training parameters and significantly lower the dimensionality of the soft prompt parameter space. It then utilizes a compressed outer product module to facilitate multiple interactions among prompt tokens, exploring their intrinsic associations to enhance knowledge representation. Finally, LAMP uses average pooling to reduce memory usage and training/inference time. Extensive experiments across six architectures and eight datasets demonstrate that LAMP outperforms state-of-the-art PT-based and LoRA-based methods in performance and efficiency.