# Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning

arXiv: https://arxiv.org/abs/2202.00535 · HTML: https://mlsanity.com/paper/2202.00535/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [A Retrieve-and-Edit Framework for Predicting Structured Outputs](https://mlsanity.com/paper/1812.01194/) (2018)
- [Paraphrase Generation with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.00279/) (2017)
- [Neural Syntactic Preordering for Controlled Paraphrase Generation](https://mlsanity.com/paper/2005.02013/) (2020)
- [A Deep Generative Framework for Paraphrase Generation](https://mlsanity.com/paper/1709.05074/) (2017)
- [Neural Paraphrase Generation with Stacked Residual LSTM Networks](https://mlsanity.com/paper/1610.03098/) (2016)
- [Syntax-guided Controlled Generation of Paraphrases](https://mlsanity.com/paper/2005.08417/) (2020)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [Controllable Paraphrase Generation with a Syntactic Exemplar](https://mlsanity.com/paper/1906.00565/) (2019)
- [Unsupervised Paraphrasing by Simulated Annealing](https://mlsanity.com/paper/1909.03588/) (2019)
- [Paraphrasing with Large Language Models](https://mlsanity.com/paper/1911.09661/) (2019)
- [Decomposable Neural Paraphrase Generation](https://mlsanity.com/paper/1906.09741/) (2019)
- [Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://mlsanity.com/paper/1810.11193/) (2018)
- [ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation](https://mlsanity.com/paper/2101.08382/) (2021)
- [Paraphrase Generation with Latent Bag of Words](https://mlsanity.com/paper/2001.01941/) (2020)

## Built on by (most cited first, 25 of 25)

- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [CoEdIT: Text Editing by Task-Specific Instruction Tuning](https://mlsanity.com/paper/2305.09857/) (2023)
- [Building Emotional Support Chatbots in the Era of LLMs](https://mlsanity.com/paper/2308.11584/) (2023)
- [ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness](https://mlsanity.com/paper/2305.12947/) (2023)
- [Continual Vision-Language Representation Learning with Off-Diagonal Information](https://mlsanity.com/paper/2305.07437/) (2023)
- [Measuring Reliability of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2211.05853/) (2022)
- [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023)
- [Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages](https://mlsanity.com/paper/2212.09651/) (2022)
- [From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL](https://mlsanity.com/paper/2311.06595/) (2023)
- [Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation](https://mlsanity.com/paper/2401.06643/) (2024)
- [mEdIT: Multilingual Text Editing via Instruction Tuning](https://mlsanity.com/paper/2402.16472/) (2024)
- [ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation](https://mlsanity.com/paper/2305.16585/) (2023)
- [Improving Tool Retrieval by Leveraging Large Language Models for Query Generation](https://mlsanity.com/paper/2412.03573/) (2024)
- [Monotonic Paraphrasing Improves Generalization of Language Model Prompting](https://mlsanity.com/paper/2403.16038/) (2024)
- [What Makes Data-to-Text Generation Hard for Pretrained Language Models?](https://mlsanity.com/paper/2205.11505/) (2022)
- [LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training](https://mlsanity.com/paper/2401.04348/) (2024)
- [LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?](https://mlsanity.com/paper/2408.16502/) (2024)
- [Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations](https://mlsanity.com/paper/2211.00881/) (2022)
- [MixPro: Simple yet Effective Data Augmentation for Prompt-based Learning](https://mlsanity.com/paper/2304.09402/) (2023)
- [Spivavtor: An Instruction Tuned Ukrainian Text Editing Model](https://mlsanity.com/paper/2404.18880/) (2024)
- [Patent-CR: A Dataset for Patent Claim Revision](https://mlsanity.com/paper/2412.02549/) (2024)
- [Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2404.12596/) (2024)
- [Learning to Adapt to Low-Resource Paraphrase Generation](https://mlsanity.com/paper/2412.17111/) (2024)
- [Task-Oriented Paraphrase Analytics](https://mlsanity.com/paper/2403.17564/) (2024)
- [Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation](https://mlsanity.com/paper/2404.05143/) (2024)

## Abstract (arXiv)

Paraphrase generation is a fundamental and long-standing task in natural language processing. In this paper, we concentrate on two contributions to the task: (1) we propose Retrieval Augmented Prompt Tuning (RAPT) as a parameter-efficient method to adapt large pre-trained language models for paraphrase generation; (2) we propose Novelty Conditioned RAPT (NC-RAPT) as a simple model-agnostic method of using specialized prompt tokens for controlled paraphrase generation with varying levels of lexical novelty. By conducting extensive experiments on four datasets, we demonstrate the effectiveness of the proposed approaches for retaining the semantic content of the original text while inducing lexical novelty in the generation.