# Unsupervised Paraphrasing with Pretrained Language Models

arXiv: https://arxiv.org/abs/2010.12885 · HTML: https://mlsanity.com/paper/2010.12885/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 15 of 15)

- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question Answering](https://mlsanity.com/paper/2305.04329/) (2023)
- [FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering](https://mlsanity.com/paper/2306.05523/) (2023)
- [BanglaParaphrase: A High-Quality Bangla Paraphrase Dataset](https://mlsanity.com/paper/2210.05109/) (2022)
- [Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models](https://mlsanity.com/paper/2305.01645/) (2023)
- [Improving Large-scale Paraphrase Acquisition and Generation](https://mlsanity.com/paper/2210.03235/) (2022)
- [Learn What NOT to Learn: Towards Generative Safety in Chatbots](https://mlsanity.com/paper/2304.11220/) (2023)
- [LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training](https://mlsanity.com/paper/2401.04348/) (2024)
- [Exploring the Landscape of Natural Language Processing Research](https://mlsanity.com/paper/2307.10652/) (2023)
- [Addressing Distribution Shift at Test Time in Pre-trained Language Models](https://mlsanity.com/paper/2212.02384/) (2022)
- [Reprint: a randomized extrapolation based on principal components for data augmentation](https://mlsanity.com/paper/2204.12024/) (2022)
- [Paraphrase Acquisition from Image Captions](https://mlsanity.com/paper/2301.11030/) (2023)
- [IdEALS: Idiomatic Expressions for Advancement of Language Skills](https://mlsanity.com/paper/2305.13637/) (2023)
- [SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation](https://mlsanity.com/paper/2507.03415/) (2025)

## Abstract (arXiv)

Paraphrase generation has benefited extensively from recent progress in the designing of training objectives and model architectures. However, previous explorations have largely focused on supervised methods, which require a large amount of labeled data that is costly to collect. To address this drawback, we adopt a transfer learning approach and propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking (DB). To enforce a surface form dissimilar from the input, whenever the language model emits a token contained in the source sequence, DB prevents the model from outputting the subsequent source token for the next generation step. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair (QQP) and the ParaNMT datasets and is robust to domain shift between the two datasets of distinct distributions. We also demonstrate that our model transfers to paraphrasing in other languages without any additional finetuning.