mlsanityData to 7 Oct 2026

Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

arXiv 2110.10668 ↗First posted 20 Oct 2021cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading ST automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify several models that correlate well with human judgments and are robust across languages. We hope that this work will help accelerate development in ST, where human evaluation is often challenging to collect.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

34 of 62 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  11. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  12. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
22 more
  1. 2020A Probabilistic Formulation of Unsupervised Text Style Transfercited by 70
  2. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  3. 2019Evaluating Style Transfer for Textcited by 59
  4. 2021XFORMAL: A Benchmark for Multilingual Formality Style Transfercited by 51
  5. 2020Parallel Data Augmentation for Formality Style Transfercited by 39
  6. 2019Reinforcement Learning Based Text Style Transfer without Parallel Training Corpuscited by 38
  7. 2020Unsupervised Text Generation by Learning from Searchcited by 36
  8. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  9. 2021StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfercited by 34
  10. 2019Domain Adaptive Text Style Transfercited by 25
  11. 2020On Learning Text Style Transfer with Direct Rewardscited by 24
  12. 2020Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metriccited by 23
  13. 2019Semi-supervised Text Style Transfer: Cross Projection in Latent Spacecited by 21
  14. 2019Style Transfer for Texts: Retrain, Report Errors, Compare with Rewritescited by 19
  15. 2021A Review of Human Evaluation for Style Transfercited by 19
  16. 2019Formality Style Transfer with Hybrid Textual Annotationscited by 16
  17. 2020Contextual Text Style Transfercited by 16
  18. 2020Exploring Contextual Word-level Style Relevance for Unsupervised Style Transfercited by 15
  19. 2020Reinforced Rewards Framework for Text Style Transfercited by 12
  20. 2015SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicaturecited by 11
  21. 2020Robust Front-End for Multi-Channel ASR using Flow-Based Density Estimationcited by 9
  22. 2019Grammatical Error Correction and Style Transfer via Zero-shot Monolingual Translationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 6; 2024: 1; 2025: 4; 2026: 4.

23 papers.

Sort:

By area:

  1. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  3. 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
  4. 2022Controlling Translation Formality Using Pre-trained Multilingual Language Modelscited by 14
  5. 2023A Call for Standardization and Validation of Text Style Transfer Evaluationcited by 13
  6. 2023Multidimensional Evaluation for Text Style Transfer Using ChatGPTcited by 12
  7. 2024ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactionscited by 12
  8. 2022Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfercited by 11
  9. 2023Text Style Transfer Evaluation Using Large Language Modelscited by 11
  10. 2022Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfercited by 8
  11. 2021From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Surveycited by 5
  12. 2025Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?cited by 4
11 more
  1. 2023In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languagescited by 2
  2. 2022Studying the role of named entities for content preservation in text style transfercited by 1
  3. 2023Responsibility Perspective Transfer for Italian Femicide Newscited by 1
  4. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 1
  5. 2025Absolute Evaluation Measures for Machine Learning: A Surveycited by 1
  6. 2025Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluationcited by 1
  7. 2026Unsupervised Text Style Transfer for Controllable Intensitycited by 1
  8. 2023Learning a Formality-Aware Japanese Sentence Representationcited by 0
  9. 2026A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEscited by 0
  10. 2026From Standard English to Singlish: A Retrieval-Augmented Approach for Code-Switched Creole Generation in Large Language Modelscited by 0
  11. 2026Style or Content? Evaluating Style Classifiers with Controlled Content Overlapcited by 0