# Paraphrase Generation with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.00279 · HTML: https://mlsanity.com/paper/1711.00279/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2017.

## Built on by (most cited first, 30 of 58)

- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Text Generation by Learning from Demonstrations](https://mlsanity.com/paper/2009.07839/) (2020)
- [No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices](https://mlsanity.com/paper/2402.16187/) (2024)
- [Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs](https://mlsanity.com/paper/2101.10579/) (2021)
- [EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling](https://mlsanity.com/paper/2403.14541/) (2024)
- [Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/2106.08942/) (2021)
- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Evading Data Contamination Detection for Language Models is (too) Easy](https://mlsanity.com/paper/2402.02823/) (2024)
- [Domain Adaptive Fake News Detection via Reinforcement Learning](https://mlsanity.com/paper/2202.08159/) (2022)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning](https://mlsanity.com/paper/2202.00535/) (2022)
- [Generative Transformers for Design Concept Generation](https://mlsanity.com/paper/2211.03468/) (2022)
- [Data Expansion using Back Translation and Paraphrasing for Hate Speech Detection](https://mlsanity.com/paper/2106.04681/) (2021)
- [A Review of Text Style Transfer using Deep Learning](https://mlsanity.com/paper/2109.15144/) (2021)
- [Generative Pre-Trained Transformer for Design Concept Generation: An Exploration](https://mlsanity.com/paper/2111.08489/) (2021)
- [Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing](https://mlsanity.com/paper/2502.15208/) (2025)
- [ConRPG: Paraphrase Generation using Contexts as Regularizer](https://mlsanity.com/paper/2109.00363/) (2021)
- [Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors](https://mlsanity.com/paper/2204.01227/) (2022)
- [Paraphrase Identification with Deep Learning: A Review of Datasets and Methods](https://mlsanity.com/paper/2212.06933/) (2022)
- [An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation](https://mlsanity.com/paper/1908.10835/) (2019)
- [Non-Parallel Text Style Transfer with Self-Parallel Supervision](https://mlsanity.com/paper/2204.08123/) (2022)
- [Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning](https://mlsanity.com/paper/2103.12777/) (2021)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Monotonic Paraphrasing Improves Generalization of Language Model Prompting](https://mlsanity.com/paper/2403.16038/) (2024)
- [Efficient Reinforcement Learning for Unsupervised Controlled Text Generation](https://mlsanity.com/paper/2204.07696/) (2022)
- [A Comprehensive Study on NLP Data Augmentation for Hate Speech Detection: Legacy Methods, BERT, and LLMs](https://mlsanity.com/paper/2404.00303/) (2024)
- [Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control](https://mlsanity.com/paper/2211.05750/) (2022)
- [Corpus-Based Paraphrase Detection Experiments and Review](https://mlsanity.com/paper/2106.00145/) (2021)
- [Simulated annealing for optimization of graphs and sequences](https://mlsanity.com/paper/2110.01384/) (2021)

## Abstract (arXiv)

Automatic generation of paraphrases from a given sentence is an important yet challenging task in natural language processing (NLP), and plays a key role in a number of applications such as question answering, search, and dialogue. In this paper, we present a deep reinforcement learning approach to paraphrase generation. Specifically, we propose a new framework for the task, which consists of a textitgenerator and an textitevaluator, both of which are learned from data. The generator, built as a sequence-to-sequence learning model, can produce paraphrases given a sentence. The evaluator, constructed as a deep matching model, can judge whether two sentences are paraphrases of each other. The generator is first trained by deep learning and then further fine-tuned by reinforcement learning in which the reward is given by the evaluator. For the learning of the evaluator, we propose two methods based on supervised learning and inverse reinforcement learning respectively, depending on the type of available training data. Empirical study shows that the learned evaluator can guide the generator to produce more accurate paraphrases. Experimental results demonstrate the proposed models (the generators) outperform the state-of-the-art methods in paraphrase generation in both automatic evaluation and human evaluation.