mlsanityData to 14 Oct 2026

Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer

arXiv 2204.07549 ↗First posted 15 Apr 2022cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Although text style transfer has witnessed rapid development in recent years, there is as yet no established standard for evaluation, which is performed using several automatic metrics, lacking the possibility of always resorting to human judgement. We focus on the task of formality transfer, and on the three aspects that are usually evaluated: style strength, content preservation, and fluency. To cast light on how such aspects are assessed by common and new metrics, we run a human-based evaluation and perform a rich correlation analysis. We are then able to offer some recommendations on the use of such metrics in formality transfer, also with an eye to their generalisability (or not) to related tasks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 29 of its 51 references that are in the map. First posted on arXiv 15 Apr 2022.

Builds on

29 of 51 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  7. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
  8. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  9. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 183
  10. 2020Deep Learning for Text Style Transfer: A Surveycited by 162
  11. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  12. 2018Style Transfer Through Back-Translationcited by 124
17 more
  1. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  2. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  3. 2019Evaluating Style Transfer for Textcited by 61
  4. 2018Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approachcited by 55
  5. 2021XFORMAL: A Benchmark for Multilingual Formality Style Transfercited by 52
  6. 2020Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymencited by 45
  7. 2021LEWIS: Levenshtein Editing for Unsupervised Text Style Transfercited by 40
  8. 2021StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfercited by 34
  9. 2018Multi-Task Neural Models for Translating Between Styles Within and Across Languagescited by 28
  10. 2021Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfercited by 28
  11. 2021Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfercited by 25
  12. 2020Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metriccited by 23
  13. 2019Style Transfer for Texts: Retrain, Report Errors, Compare with Rewritescited by 20
  14. 2021Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalizationcited by 20
  15. 2020Cycle-Consistent Adversarial Autoencoders for Unsupervised Text Style Transfercited by 18
  16. 2020Exploring Contextual Word-level Style Relevance for Unsupervised Style Transfercited by 15
  17. 2021Generic resources are what you need: Style transfer tasks without task-specific parallel training datacited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 1; 2025: 3; 2026: 1.

12 papers.

  1. 2023Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxificationcited by 15
  2. 2023Multidimensional Evaluation for Text Style Transfer Using ChatGPTcited by 12
  3. 2022Detecting Text Formality: A Study of Text Classification Approachescited by 11
  4. 2023Pre-Trained Language-Meaning Models for Multilingual Parsing and Generationcited by 10
  5. 2023RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translationcited by 9
  6. 2024LMStyle Benchmark: Evaluating Text Style Transfer for Chatbotscited by 5
  7. 2022Multi-Figurative Language Generationcited by 2
  8. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 2
  9. 2023Responsibility Perspective Transfer for Italian Femicide Newscited by 1
  10. 2025IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Datacited by 1
  11. 2025Adaptive Originality Filtering: Rejection Based Prompting and RiddleScore for Culturally Grounded Multilingual Riddle Generationcited by 1
  12. 2026Metric-Construction Coupling Inflates Measured Synthetic Dialect Recoverycited by 0