# Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models

arXiv: https://arxiv.org/abs/2310.01691 · HTML: https://mlsanity.com/paper/2310.01691/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2303.02861/) (2023)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [On Transferability of Prompt Tuning for Natural Language Processing](https://mlsanity.com/paper/2111.06719/) (2021)
- [Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts](https://mlsanity.com/paper/2112.08348/) (2021)
- [ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training](https://mlsanity.com/paper/2210.01738/) (2022)
- [A Closer Look at How Fine-tuning Changes BERT](https://mlsanity.com/paper/2106.14282/) (2021)
- [Can discrete information extraction prompts generalize across language models?](https://mlsanity.com/paper/2302.09865/) (2023)
- [The Emergence of Chunking Structures with Hierarchical RNN](https://mlsanity.com/paper/2309.04919/) (2023)

## Built on by (most cited first, 7 of 7)

- [Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models](https://mlsanity.com/paper/2506.06844/) (2025)
- [TokMem: One-Token Procedural Memory for Large Language Models](https://mlsanity.com/paper/2510.00444/) (2025)
- [LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades](https://mlsanity.com/paper/2505.13515/) (2025)
- [Order-Level Attention Similarity Across Language Models: A Latent Commonality](https://mlsanity.com/paper/2511.05064/) (2025)
- [Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation](https://mlsanity.com/paper/2405.16895/) (2024)
- [Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs](https://mlsanity.com/paper/2506.16196/) (2025)
- [Understanding Visual Concepts Across Models](https://mlsanity.com/paper/2406.07506/) (2024)

## Abstract (arXiv)

Prompt tuning in natural language processing (NLP) has become an increasingly popular method for adapting large language models to specific tasks. However, the transferability of these prompts, especially continuous prompts, between different models remains a challenge. In this work, we propose a zero-shot continuous prompt transfer method, where source prompts are encoded into relative space and the corresponding target prompts are searched for transferring to target models. Experimental results confirm the effectiveness of our method, showing that 'task semantics' in continuous prompts can be generalized across various language models. Moreover, we find that combining 'task semantics' from multiple source models can further enhance the generalizability of transfer.