mlsanityData to 14 Oct 2026

An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation

arXiv 1908.10835 ↗First posted 28 Aug 2019cs.CL cs.LGcited by 11

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Generating paraphrases from given sentences involves decoding words step by step from a large vocabulary. To learn a decoder, supervised learning which maximizes the likelihood of tokens always suffers from the exposure bias. Although both reinforcement learning (RL) and imitation learning (IL) have been widely used to alleviate the bias, the lack of direct comparison leads to only a partial image on their benefits. In this work, we present an empirical study on how RL and IL can help boost the performance of generating paraphrases, with the pointer-generator as a base model. Experiments on the benchmark datasets show that (1) imitation learning is constantly better than reinforcement learning; and (2) the pointer-generator models with imitation learning outperform the state-of-the-art methods with a large margin.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 40 references that are in the map. First posted on arXiv 28 Aug 2019.

Builds on

19 of 40 references.

References · 19

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  6. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  7. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
  8. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  9. 2015Minimum Risk Training for Neural Machine Translationcited by 115
  10. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  11. 2017A Continuously Growing Dataset of Sentential Paraphrasescited by 72
  12. 2017Paraphrase Generation with Deep Reinforcement Learningcited by 58
7 more
  1. 2016Neural Paraphrase Generation with Stacked Residual LSTM Networkscited by 52
  2. 2015Learning to Search Better Than Your Teachercited by 50
  3. 2017Learning to Paraphrase for Question Answeringcited by 46
  4. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  5. 2018Deep Reinforcement Learning For Sequence to Sequence Modelscited by 39
  6. 2015Describing Multimedia Content using Attention-based Encoder--Decoder Networkscited by 32
  7. 2018Deep Reinforcement Learning for Chinese Zero pronoun Resolutioncited by 5

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 5; 2023: 2; 2024: 2.

11 papers.

  1. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  2. 2022Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuningcited by 25
  3. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  4. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  5. 2021Constrained Text Generation with Global Guidance -- Case Study on CommonGencited by 10
  6. 2023Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translationcited by 2
  7. 2024RIFF: Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Modelscited by 2
  8. 2021Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translationcited by 1
  9. 2022Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systemscited by 0
  10. 2022Fine-Grained Emotional Paraphrasing along Emotion Gradientscited by 0
  11. 2023Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Text Generation by Learning from Demonstrations
  2. 2024Improved Paraphrase Generation via Controllable Latent Diffusion
  3. 2024Imitating Language via Scalable Inverse Reinforcement Learning
  4. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  6. 2023Vector-Quantized Prompt Learning for Paraphrase Generation
  7. 2025On a Connection Between Imitation Learning and RLHF
  8. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generation