# Reducing Retraining by Recycling Parameter-Efficient Prompts

arXiv: https://arxiv.org/abs/2208.05577 · HTML: https://mlsanity.com/paper/2208.05577/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 10 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://mlsanity.com/paper/1810.12885/) (2018)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [Statistically Significant Detection of Linguistic Change](https://mlsanity.com/paper/1411.3315/) (2014)
- [Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts](https://mlsanity.com/paper/2112.08348/) (2021)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)

## Built on by (most cited first, 11 of 11)

- [VPGTrans: Transfer Visual Prompt Generator across LLMs](https://mlsanity.com/paper/2305.01278/) (2023)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)
- [ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2311.16494/) (2023)
- [Training LLMs over Neurally Compressed Text](https://mlsanity.com/paper/2404.03626/) (2024)
- [Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models](https://mlsanity.com/paper/2306.04529/) (2023)
- [Extensible Prompts for Language Models on Zero-shot Language Style Customization](https://mlsanity.com/paper/2212.00616/) (2022)
- [Efficient Model Development through Fine-tuning Transfer](https://mlsanity.com/paper/2503.20110/) (2025)
- [Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models](https://mlsanity.com/paper/2506.06844/) (2025)
- [LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades](https://mlsanity.com/paper/2505.13515/) (2025)

## Abstract (arXiv)

Parameter-efficient methods are able to use a single frozen pre-trained large language model (LLM) to perform many tasks by learning task-specific soft prompts that modulate model behavior when concatenated to the input text. However, these learned prompts are tightly coupled to a given frozen model -- if the model is updated, corresponding new prompts need to be obtained. In this work, we propose and investigate several approaches to "Prompt Recycling'" where a prompt trained on a source model is transformed to work with the new target model. Our methods do not rely on supervised pairs of prompts, task-specific data, or training updates with the target model, which would be just as costly as re-tuning prompts with the target model from scratch. We show that recycling between models is possible (our best settings are able to successfully recycle 88.9% of prompts, producing a prompt that out-performs baselines), but significant performance headroom remains, requiring improved recycling techniques.