# Ultra-Low-Dimensional Prompt Tuning via Random Projection

arXiv: https://arxiv.org/abs/2502.04501 · HTML: https://mlsanity.com/paper/2502.04501/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2025.

## Built on by (most cited first, 2 of 2)

- [TokMem: One-Token Procedural Memory for Large Language Models](https://mlsanity.com/paper/2510.00444/) (2025)
- [Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling](https://mlsanity.com/paper/2604.04987/) (2026)

## Abstract (arXiv)

Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parameter-efficiency by learning prompt embeddings, but these embeddings are typically tied to the model's hidden dimensionality, limiting parameter saving. In this paper, we propose Ultra-Low-dimensional Prompt Tuning (ULPT), a simple yet effective method that optimizes prompts in a low-dimensional space (e.g., 2D) and uses a frozen random matrix for up-projection. ULPT can achieve 98% reduction in the training parameters compared to vanilla prompt tuning while preserving performance. Our extensive experiments across over 20 NLP tasks demonstrate that ULPT consistently outperforms recent parameter-efficient tuning methods using significantly fewer parameters, making it well-suited as a storage-efficient framework for massive LLM customization.