# Zero-Shot Paraphrase Generation with Multilingual Language Models

arXiv: https://arxiv.org/abs/1911.03597 · HTML: https://mlsanity.com/paper/1911.03597/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2019.

## Built on by (most cited first, 7 of 7)

- [LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training](https://mlsanity.com/paper/2401.04348/) (2024)
- [MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion](https://mlsanity.com/paper/2212.09666/) (2022)
- [A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages](https://mlsanity.com/paper/2308.09435/) (2023)
- [Improving the Diversity of Unsupervised Paraphrasing with Embedding Outputs](https://mlsanity.com/paper/2110.13231/) (2021)
- [FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models](https://mlsanity.com/paper/2206.06586/) (2022)
- [Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision](https://mlsanity.com/paper/2106.05166/) (2021)
- [A Single Model Ensemble Framework for Neural Machine Translation using Pivot Translation](https://mlsanity.com/paper/2502.01182/) (2025)

## Abstract (arXiv)

Leveraging multilingual parallel texts to automatically generate paraphrases has drawn much attention as size of high-quality paraphrase corpus is limited. Round-trip translation, also known as the pivoting method, is a typical approach to this end. However, we notice that the pivoting process involves multiple machine translation models and is likely to incur semantic drift during the two-step translations. In this paper, inspired by the Transformer-based language models, we propose a simple and unified paraphrasing model, which is purely trained on multilingual parallel data and can conduct zero-shot paraphrase generation in one step. Compared with the pivoting approach, paraphrases generated by our model is more semantically similar to the input sentence. Moreover, since our model shares the same architecture as GPT (Radford et al., 2018), we are able to pre-train the model on large-scale unparallel corpus, which further improves the fluency of the output sentences. In addition, we introduce the mechanism of denoising auto-encoder (DAE) to improve diversity and robustness of the model. Experimental results show that our model surpasses the pivoting method in terms of relevance, diversity, fluency and efficiency.