mlsanityData to 14 Oct 2026

A Review of Human Evaluation for Style Transfer

arXiv 2106.04747 ↗First posted 9 Jun 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

This paper reviews and summarizes human evaluation practices described in 97 style transfer papers with respect to three main evaluation aspects: style transfer, meaning preservation, and fluency. In principle, evaluations by human raters should be the most reliable. However, in style transfer papers, we find that protocols for human evaluations are often underspecified and not standardized, which hampers the reproducibility of research in this field and progress toward better human and automatic evaluation methods.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 17 of its 37 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

17 of 37 references.

References · 17

  1. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  2. 2016A Persona-Based Neural Conversation Modelcited by 223
  3. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
  4. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  5. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 183
  6. 2020Deep Learning for Text Style Transfer: A Surveycited by 162
  7. 2019Automatically Neutralizing Subjective Bias in Textcited by 105
  8. 2019Evaluating Style Transfer for Textcited by 61
  9. 2020PowerTransformer: Unsupervised Controllable Revision for Biased Language Correctioncited by 59
  10. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 53
  11. 2021XFORMAL: A Benchmark for Multilingual Formality Style Transfercited by 52
  12. 2021A Systematic Review of Reproducibility Research in Natural Language Processingcited by 34
5 more
  1. 2019IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translationcited by 33
  2. 2019Domain Adaptive Text Style Transfercited by 25
  3. 2021The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLPcited by 19
  4. 2020A Survey on Text Simplificationcited by 13
  5. 2021The Great Misalignment Problem in Human Evaluation of NLP Methodscited by 8

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 6; 2023: 5; 2024: 0; 2025: 3.

19 papers.

Sort:

By area:

  1. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  2. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  3. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  4. 2023Training Models to Generate, Recognize, and Reframe Unhelpful Thoughtscited by 31
  5. 2021Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfercited by 25
  6. 2021Few-shot Controllable Style Transfer for Low-Resource Multilingual Settingscited by 21
  7. 2022FRMT: A Benchmark for Few-Shot Region-Aware Machine Translationcited by 18
  8. 2023Prompt-Based Editing for Text Style Transfercited by 16
  9. 2023Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSAcited by 15
  10. 2023A Call for Standardization and Validation of Text Style Transfer Evaluationcited by 14
  11. 2023Text Style Transfer Evaluation Using Large Language Modelscited by 13
  12. 2025Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?cited by 7
7 more
  1. 2021From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Surveycited by 5
  2. 2022StyleFlow: Disentangle Latent Representations via Normalizing Flow for Unsupervised Text Style Transfercited by 4
  3. 2025Evaluating Style-Personalized Text Generation: Challenges and Directionscited by 3
  4. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 2
  5. 2021Contextualizing Variation in Text Style Transfer Datasetscited by 1
  6. 2022Studying the role of named entities for content preservation in text style transfercited by 1
  7. 2022A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deceptioncited by 1