# Learning to Transfer Prompts for Text Generation

arXiv: https://arxiv.org/abs/2205.01543 · HTML: https://mlsanity.com/paper/2205.01543/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [Bilateral Multi-Perspective Matching for Natural Language Sentences](https://mlsanity.com/paper/1702.03814/) (2017)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)

## Built on by (most cited first, 22 of 22)

- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2309.05173/) (2023)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization](https://mlsanity.com/paper/2211.09783/) (2022)
- [Controlling Personality Style in Dialogue with Zero-Shot Prompt-Based Learning](https://mlsanity.com/paper/2302.03848/) (2023)
- [Effective Structured Prompting by Meta-Learning and Representative Verbalizer](https://mlsanity.com/paper/2306.00618/) (2023)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)
- [ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation](https://mlsanity.com/paper/2210.13304/) (2022)
- [PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learners](https://mlsanity.com/paper/2205.09229/) (2022)
- [Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4](https://mlsanity.com/paper/2407.04130/) (2024)
- [Enhancing Cross-task Transfer of Large Language Models via Activation Steering](https://mlsanity.com/paper/2507.13236/) (2025)
- [Relational Prompt-based Pre-trained Language Models for Social Event Detection](https://mlsanity.com/paper/2404.08263/) (2024)
- [ADEPT: Adaptive Dynamic Early-Exit Process for Transformers](https://mlsanity.com/paper/2601.03700/) (2026)
- [TextBox 2.0: A Text Generation Library with Pre-trained Language Models](https://mlsanity.com/paper/2212.13005/) (2022)
- [Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator](https://mlsanity.com/paper/2403.12407/) (2024)
- [On Conditional and Compositional Language Model Differentiable Prompting](https://mlsanity.com/paper/2307.01446/) (2023)
- [Palette of Language Models: A Solver for Controlled Text Generation](https://mlsanity.com/paper/2503.11182/) (2025)
- [Materials Transformers Language Models for Generative Materials Design: a benchmark study](https://mlsanity.com/paper/2206.13578/) (2022)
- [Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases](https://mlsanity.com/paper/2403.16396/) (2024)
- [SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning](https://mlsanity.com/paper/2407.15556/) (2024)

## Abstract (arXiv)

Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning. While, it is challenging to fine-tune PLMs in a data-scarce situation. Therefore, it is non-trivial to develop a general and lightweight model that can adapt to various text generation tasks based on PLMs. To fulfill this purpose, the recent prompt-based learning offers a potential solution. In this paper, we improve this technique and propose a novel prompt-based method (PTG) for text generation in a transferable setting. First, PTG learns a set of source prompts for various source generation tasks and then transfers these prompts as target prompts to perform target generation tasks. To consider both task- and instance-level information, we design an adaptive attention mechanism to derive the target prompts. For each data instance, PTG learns a specific target prompt by attending to highly relevant source prompts. In extensive experiments, PTG yields competitive or better results than fine-tuning methods. We release our source prompts as an open resource, where users can add or reuse them to improve new text generation tasks for future research. Code and data can be available at https://github.com/RUCAIBox/Transfer-Prompts-for-Text-Generation.