# NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer

arXiv: https://arxiv.org/abs/2106.02210 · HTML: https://mlsanity.com/paper/2106.02210/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://mlsanity.com/paper/1808.04339/) (2018)
- [Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation](https://mlsanity.com/paper/1905.05621/) (2019)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [A Probabilistic Formulation of Unsupervised Text Style Transfer](https://mlsanity.com/paper/2002.03912/) (2020)
- [A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1905.10060/) (2019)
- [Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Models](https://mlsanity.com/paper/1707.01161/) (2017)
- ["Mask and Infill" : Applying Masked Language Model to Sentiment Transfer](https://mlsanity.com/paper/1908.08039/) (2019)
- [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018)
- [Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information](https://mlsanity.com/paper/1911.02215/) (2019)
- [ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2005.00850/) (2020)
- [A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1906.01833/) (2019)
- [Non-autoregressive Transformer by Position Learning](https://mlsanity.com/paper/1911.10677/) (2019)
- [Exploring Contextual Word-level Style Relevance for Unsupervised Style Transfer](https://mlsanity.com/paper/2005.02049/) (2020)
- [Non-Autoregressive Neural Dialogue Generation](https://mlsanity.com/paper/2002.04250/) (2020)
- [Robust Front-End for Multi-Channel ASR using Flow-Based Density Estimation](https://mlsanity.com/paper/2007.12903/) (2020)
- [Challenges in Emotion Style Transfer: An Exploration with a Lexical Substitution Pipeline](https://mlsanity.com/paper/2005.07617/) (2020)

## Built on by (most cited first, 7 of 7)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain](https://mlsanity.com/paper/2305.03256/) (2023)
- [StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing](https://mlsanity.com/paper/2208.13423/) (2022)
- [Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation](https://mlsanity.com/paper/2304.11791/) (2023)
- [SC2: Towards Enhancing Content Preservation and Style Consistency in Long Text Style Transfer](https://mlsanity.com/paper/2406.04578/) (2024)
- [Replacing Language Model for Style Transfer](https://mlsanity.com/paper/2211.07343/) (2022)
- [A Survey of Text Style Transfer: Applications and Ethical Implications](https://mlsanity.com/paper/2407.16737/) (2024)

## Abstract (arXiv)

Autoregressive models have been widely used in unsupervised text style transfer. Despite their success, these models still suffer from the content preservation problem that they usually ignore part of the source sentence and generate some irrelevant words with strong styles. In this paper, we propose a Non-Autoregressive generator for unsupervised text Style Transfer (NAST), which alleviates the problem from two aspects. First, we observe that most words in the transferred sentence can be aligned with related words in the source sentence, so we explicitly model word alignments to suppress irrelevant words. Second, existing models trained with the cycle loss align sentences in two stylistic text spaces, which lacks fine-grained control at the word level. The proposed non-autoregressive generator focuses on the connections between aligned words, which learns the word-level transfer between styles. For experiments, we integrate the proposed generator into two base models and evaluate them on two style transfer tasks. The results show that NAST can significantly improve the overall performance and provide explainable word alignments. Moreover, the non-autoregressive generator achieves over 10x speedups at inference. Our codes are available at https://github.com/thu-coai/NAST.