mlsanityData to 7 Oct 2026

A Call for Standardization and Validation of Text Style Transfer Evaluation

arXiv 2306.00539 ↗First posted 1 Jun 2023cs.LG cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Text Style Transfer (TST) evaluation is, in practice, inconsistent. Therefore, we conduct a meta-analysis on human and automated TST evaluation and experimentation that thoroughly examines existing literature in the field. The meta-analysis reveals a substantial standardization gap in human and automated evaluation. In addition, we also find a validation gap: only few automated metrics have been validated using human experiments. To this end, we thoroughly scrutinize both the standardization and validation gap and reveal the resulting pitfalls. This work also paves the way to close the standardization and validation gap in TST evaluation by calling out requirements to be met by future research.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 97 of its 168 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

97 of 168 references.

References · 97

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
38 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  5. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  6. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  7. 2020The Pushshift Reddit Datasetcited by 289
  8. 2017Toward Controlled Generation of Textcited by 271
  9. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 246
  10. 2016A Persona-Based Neural Conversation Modelcited by 221
  11. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  12. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  13. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  14. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  15. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  16. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
  17. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  18. 2020With Little Power Comes Great Responsibilitycited by 138
  19. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  20. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  21. 2018Style Transfer Through Back-Translationcited by 124
  22. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  23. 2019Automatically Neutralizing Subjective Bias in Textcited by 105
  24. 2017Generating Sentences by Editing Prototypescited by 88
  25. 2019Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representationcited by 84
  26. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  27. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  28. 2020Politeness Transfer: A Tag and Generate Approachcited by 78
  29. 2018Unsupervised Text Style Transfer using Language Models as Discriminatorscited by 70
  30. 2020A Probabilistic Formulation of Unsupervised Text Style Transfercited by 70
  31. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  32. 2019Evaluating Style Transfer for Textcited by 59
  33. 2020PowerTransformer: Unsupervised Controllable Revision for Biased Language Correctioncited by 59
  34. 2018Polite Dialogue Generation Without Parallel Datacited by 57
  35. 2018Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approachcited by 55
  36. 2018Fighting Offensive Language on Social Media with Unsupervised Text Style Transfercited by 55
  37. 2018Engaging Image Captioning Via Personalitycited by 51
  38. 2021XFORMAL: A Benchmark for Multilingual Formality Style Transfercited by 51

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024On the Challenges and Opportunities in Generative AIcited by 31
  2. 2023Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxificationcited by 15
  3. 2024Distilling Text Style Transfer With Self-Explanation From LLMscited by 12
  4. 2023Text Style Transfer Evaluation Using Large Language Modelscited by 11
  5. 2024DetoxLLM: A Framework for Detoxification with Explanationscited by 10
  6. 2023Evaluating Dynamic Topic Modelscited by 4
  7. 2025Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMscited by 3
  8. 2024Tethering Broken Themes: Aligning Neural Topic Models with Labels and Authorscited by 1
  9. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 1
  10. 2025DiffStyleTS: Diffusion Model for Style Transfer in Time Seriescited by 1
  11. 2024SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learningcited by 0
  12. 2025Challenging Assumptions in Learning Generic Text Style Embeddingscited by 0
1 more
  1. 2026Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Datasetcited by 0