# A Review of Human Evaluation for Style Transfer

arXiv: https://arxiv.org/abs/2106.04747 · HTML: https://mlsanity.com/paper/2106.04747/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [Automatically Neutralizing Subjective Bias in Text](https://mlsanity.com/paper/1911.09709/) (2019)
- [Evaluating Style Transfer for Text](https://mlsanity.com/paper/1904.02295/) (2019)
- [PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction](https://mlsanity.com/paper/2010.13816/) (2020)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [XFORMAL: A Benchmark for Multilingual Formality Style Transfer](https://mlsanity.com/paper/2104.04108/) (2021)
- [A Systematic Review of Reproducibility Research in Natural Language Processing](https://mlsanity.com/paper/2103.07929/) (2021)
- [IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation](https://mlsanity.com/paper/1901.11333/) (2019)
- [Domain Adaptive Text Style Transfer](https://mlsanity.com/paper/1908.09395/) (2019)
- [The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP](https://mlsanity.com/paper/2103.09710/) (2021)
- [A Survey on Text Simplification](https://mlsanity.com/paper/2008.08612/) (2020)
- [The Great Misalignment Problem in Human Evaluation of NLP Methods](https://mlsanity.com/paper/2104.05361/) (2021)

## Built on by (most cited first, 19 of 19)

- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://mlsanity.com/paper/2212.09739/) (2022)
- [Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts](https://mlsanity.com/paper/2307.02768/) (2023)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings](https://mlsanity.com/paper/2110.07385/) (2021)
- [FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation](https://mlsanity.com/paper/2210.00193/) (2022)
- [Prompt-Based Editing for Text Style Transfer](https://mlsanity.com/paper/2301.11997/) (2023)
- [Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA](https://mlsanity.com/paper/2305.14458/) (2023)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [Text Style Transfer Evaluation Using Large Language Models](https://mlsanity.com/paper/2308.13577/) (2023)
- [Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?](https://mlsanity.com/paper/2502.04718/) (2025)
- [From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey](https://mlsanity.com/paper/2110.15871/) (2021)
- [StyleFlow: Disentangle Latent Representations via Normalizing Flow for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2212.09670/) (2022)
- [Evaluating Style-Personalized Text Generation: Challenges and Directions](https://mlsanity.com/paper/2508.06374/) (2025)
- [Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics](https://mlsanity.com/paper/2502.15022/) (2025)
- [Contextualizing Variation in Text Style Transfer Datasets](https://mlsanity.com/paper/2108.07871/) (2021)
- [Studying the role of named entities for content preservation in text style transfer](https://mlsanity.com/paper/2206.09676/) (2022)
- [A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception](https://mlsanity.com/paper/2208.05757/) (2022)

## Abstract (arXiv)

This paper reviews and summarizes human evaluation practices described in 97 style transfer papers with respect to three main evaluation aspects: style transfer, meaning preservation, and fluency. In principle, evaluations by human raters should be the most reliable. However, in style transfer papers, we find that protocols for human evaluations are often underspecified and not standardized, which hampers the reproducibility of research in this field and progress toward better human and automatic evaluation methods.