# Evaluating Style Transfer for Text

arXiv: https://arxiv.org/abs/1904.02295 · HTML: https://mlsanity.com/paper/1904.02295/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 30 of 61)

- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://mlsanity.com/paper/2212.09739/) (2022)
- [XFORMAL: A Benchmark for Multilingual Formality Style Transfer](https://mlsanity.com/paper/2104.04108/) (2021)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models](https://mlsanity.com/paper/2205.11503/) (2022)
- [StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer](https://mlsanity.com/paper/2104.05196/) (2021)
- [Inducing Positive Perspectives with Text Reframing](https://mlsanity.com/paper/2204.02952/) (2022)
- [Beyond the Surface: Measuring Self-Preference in LLM Judgments](https://mlsanity.com/paper/2506.02592/) (2025)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings](https://mlsanity.com/paper/2110.07385/) (2021)
- [A Review of Human Evaluation for Style Transfer](https://mlsanity.com/paper/2106.04747/) (2021)
- [Translating Across Cultures: LLMs for Intralingual Cultural Adaptation](https://mlsanity.com/paper/2406.14504/) (2024)
- [Disentangling Representations of Text by Masking Transformers](https://mlsanity.com/paper/2104.07155/) (2021)
- [Boosting Distress Support Dialogue Responses with Motivational Interviewing Strategy](https://mlsanity.com/paper/2305.10195/) (2023)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [A Survey of Natural Language Generation](https://mlsanity.com/paper/2112.11739/) (2021)
- [Text Style Transfer Evaluation Using Large Language Models](https://mlsanity.com/paper/2308.13577/) (2023)
- [Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer](https://mlsanity.com/paper/2204.07549/) (2022)
- [ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactions](https://mlsanity.com/paper/2407.02472/) (2024)
- [Non-Parallel Text Style Transfer with Self-Parallel Supervision](https://mlsanity.com/paper/2204.08123/) (2022)
- [Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain](https://mlsanity.com/paper/2305.03256/) (2023)
- [Text Detoxification as Style Transfer in English and Hindi](https://mlsanity.com/paper/2402.07767/) (2024)
- [Multilingual Text Style Transfer: Datasets & Models for Indian Languages](https://mlsanity.com/paper/2405.20805/) (2024)
- [Mitigating Toxic Degeneration with Empathetic Data: Exploring the Relationship Between Toxicity and Empathy](https://mlsanity.com/paper/2205.07233/) (2022)
- [Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2205.02309/) (2022)
- [Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?](https://mlsanity.com/paper/2502.04718/) (2025)
- [Masks and Mimicry: Strategic Obfuscation and Impersonation Attacks on Authorship Verification](https://mlsanity.com/paper/2503.19099/) (2025)

## Abstract (arXiv)

Research in the area of style transfer for text is currently bottlenecked by a lack of standard evaluation practices. This paper aims to alleviate this issue by experimentally identifying best practices with a Yelp sentiment dataset. We specify three aspects of interest (style transfer intensity, content preservation, and naturalness) and show how to obtain more reliable measures of them from human evaluation than in previous work. We propose a set of metrics for automated evaluation and demonstrate that they are more strongly correlated and in agreement with human judgment: direction-corrected Earth Mover's Distance, Word Mover's Distance on style-masked texts, and adversarial classification for the respective aspects. We also show that the three examined models exhibit tradeoffs between aspects of interest, demonstrating the importance of evaluating style transfer models at specific points of their tradeoff plots. We release software with our evaluation metrics to facilitate research.