# Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?

arXiv: https://arxiv.org/abs/2502.04718 · HTML: https://mlsanity.com/paper/2502.04718/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Politeness Transfer: A Tag and Generate Approach](https://mlsanity.com/paper/2004.14257/) (2020)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [Evaluating Style Transfer for Text](https://mlsanity.com/paper/1904.02295/) (2019)
- [Re-evaluating Automatic Metrics for Image Captioning](https://mlsanity.com/paper/1612.07600/) (2016)
- [SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features](https://mlsanity.com/paper/2206.07023/) (2022)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric](https://mlsanity.com/paper/2004.05001/) (2020)
- [Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites](https://mlsanity.com/paper/1908.06809/) (2019)
- [A Review of Human Evaluation for Style Transfer](https://mlsanity.com/paper/2106.04747/) (2021)
- [TwistList: Resources and Baselines for Tongue Twister Generation](https://mlsanity.com/paper/2306.03457/) (2023)
- [Are Large Language Models Actually Good at Text Style Transfer?](https://mlsanity.com/paper/2406.05885/) (2024)
- [Text Detoxification as Style Transfer in English and Hindi](https://mlsanity.com/paper/2402.07767/) (2024)
- [Rethinking Text Attribute Transfer: A Lexical Analysis](https://mlsanity.com/paper/1909.12335/) (2019)
- [Multilingual Text Style Transfer: Datasets & Models for Indian Languages](https://mlsanity.com/paper/2405.20805/) (2024)
- [Balancing the Style-Content Trade-Off in Sentiment Transfer Using Polarity-Aware Denoising](https://mlsanity.com/paper/2312.14708/) (2023)
- [Text Style Transfer: An Introductory Overview](https://mlsanity.com/paper/2407.14822/) (2024)
- [SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluation](https://mlsanity.com/paper/2206.10029/) (2022)

## Built on by (most cited first, 7 of 7)

- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [Unsupervised Text Style Transfer for Controllable Intensity](https://mlsanity.com/paper/2601.01060/) (2026)
- [Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages](https://mlsanity.com/paper/2607.02235/) (2026)
- [Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset](https://mlsanity.com/paper/2605.29365/) (2026)
- [Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery](https://mlsanity.com/paper/2610.00164/) (2026)
- [Anchors Enhance Controllability and Reasoning of Diffusion Language Models](https://mlsanity.com/paper/2610.15097/) (2026)
- [Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation](https://mlsanity.com/paper/2610.19166/) (2026)

## Abstract (arXiv)

Text style transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content. Evaluating TST outputs is a multidimensional challenge, requiring the assessment of style transfer accuracy, content preservation, and naturalness. Using human evaluation is ideal but costly, as is common in other natural language processing (NLP) tasks, however, automatic metrics for TST have not received as much attention as metrics for, e.g., machine translation or summarization. In this paper, we examine both set of existing and novel metrics from broader NLP tasks for TST evaluation, focusing on two popular subtasks, sentiment transfer and detoxification, in a multilingual context comprising English, Hindi, and Bengali. By conducting meta-evaluation through correlation with human judgments, we demonstrate the effectiveness of these metrics when used individually and in ensembles. Additionally, we investigate the potential of large language models (LLMs) as tools for TST evaluation. Our findings highlight newly applied advanced NLP metrics and LLM-based evaluations provide better insights than existing TST metrics. Our oracle ensemble approaches show even more potential.