# Unsupervised Paraphrase Generation using Pre-trained Language Models

arXiv: https://arxiv.org/abs/2006.05477 · HTML: https://mlsanity.com/paper/2006.05477/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning](https://mlsanity.com/paper/2305.18169/) (2023)
- [ConRPG: Paraphrase Generation using Contexts as Regularizer](https://mlsanity.com/paper/2109.00363/) (2021)
- [Idiomatic Expression Paraphrasing without Strong Supervision](https://mlsanity.com/paper/2112.08592/) (2021)
- [FormaT5: Abstention and Examples for Conditional Table Formatting with Natural Language](https://mlsanity.com/paper/2310.17306/) (2023)
- [Revision for Concision: A Constrained Paraphrase Generation Task](https://mlsanity.com/paper/2210.14257/) (2022)
- [Fine-Tuning Language Models for Scientific Writing Support](https://mlsanity.com/paper/2306.10974/) (2023)
- [Understanding Metrics for Paraphrasing](https://mlsanity.com/paper/2205.13119/) (2022)
- [Towards Fine-Dining Recipe Generation with Generative Pre-trained Transformers](https://mlsanity.com/paper/2209.12774/) (2022)
- [Coherence and Diversity through Noise: Self-Supervised Paraphrase Generation via Structure-Aware Denoising](https://mlsanity.com/paper/2302.02780/) (2023)
- [Paraphrase Generation as Unsupervised Machine Translation](https://mlsanity.com/paper/2109.02950/) (2021)
- [IdEALS: Idiomatic Expressions for Advancement of Language Skills](https://mlsanity.com/paper/2305.13637/) (2023)
- [Automated Annotation of Scientific Texts for ML-based Keyphrase Extraction and Validation](https://mlsanity.com/paper/2311.05042/) (2023)
- [SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation](https://mlsanity.com/paper/2507.03415/) (2025)

## Abstract (arXiv)

Large scale Pre-trained Language Models have proven to be very powerful approach in various Natural language tasks. OpenAI's GPT-2 citeradford2019language is notable for its capability to generate fluent, well formulated, grammatically consistent text and for phrase completions. In this paper we leverage this generation capability of GPT-2 to generate paraphrases without any supervision from labelled data. We examine how the results compare with other supervised and unsupervised approaches and the effect of using paraphrases for data augmentation on downstream tasks such as classification. Our experiments show that paraphrases generated with our model are of good quality, are diverse and improves the downstream task performance when used for data augmentation.