# Text Style Transfer Evaluation Using Large Language Models

arXiv: https://arxiv.org/abs/2308.13577 · HTML: https://mlsanity.com/paper/2308.13577/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective](https://mlsanity.com/paper/2010.02329/) (2020)
- [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018)
- [Evaluating Style Transfer for Text](https://mlsanity.com/paper/1904.02295/) (2019)
- [Polite Dialogue Generation Without Parallel Data](https://mlsanity.com/paper/1805.03162/) (2018)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric](https://mlsanity.com/paper/2004.05001/) (2020)
- [A Review of Human Evaluation for Style Transfer](https://mlsanity.com/paper/2106.04747/) (2021)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [Probing the Robustness of Trained Metrics for Conversational Dialogue Systems](https://mlsanity.com/paper/2202.13887/) (2022)
- [On the Effectiveness of Automated Metrics for Text Generation Systems](https://mlsanity.com/paper/2210.13025/) (2022)

## Built on by (most cited first, 13 of 13)

- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [Beyond the Surface: Measuring Self-Preference in LLM Judgments](https://mlsanity.com/paper/2506.02592/) (2025)
- [Style-Specific Neurons for Steering LLMs in Text Style Transfer](https://mlsanity.com/paper/2410.00593/) (2024)
- [FairMonitor: A Dual-framework for Detecting Stereotypes and Biases in Large Language Models](https://mlsanity.com/paper/2405.03098/) (2024)
- [Evaluating Style-Personalized Text Generation: Challenges and Directions](https://mlsanity.com/paper/2508.06374/) (2025)
- [Tethering Broken Themes: Aligning Neural Topic Models with Labels and Authors](https://mlsanity.com/paper/2410.18140/) (2024)
- [HiMeS: Hippocampus-inspired Memory System for Personalized AI Assistants](https://mlsanity.com/paper/2601.06152/) (2026)
- [SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning](https://mlsanity.com/paper/2407.15556/) (2024)
- [Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responses](https://mlsanity.com/paper/2605.27971/) (2026)
- [Small Is Enough: Per-User Style Rewriting of AI-Edited Text via LoRA Adapters](https://mlsanity.com/paper/2607.29238/) (2026)
- [Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery](https://mlsanity.com/paper/2610.00164/) (2026)
- [Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation](https://mlsanity.com/paper/2610.19166/) (2026)

## Abstract (arXiv)

Evaluating Text Style Transfer (TST) is a complex task due to its multifaceted nature. The quality of the generated text is measured based on challenging factors, such as style transfer accuracy, content preservation, and overall fluency. While human evaluation is considered to be the gold standard in TST assessment, it is costly and often hard to reproduce. Therefore, automated metrics are prevalent in these domains. Nevertheless, it remains unclear whether these automated metrics correlate with human evaluations. Recent strides in Large Language Models (LLMs) have showcased their capacity to match and even exceed average human performance across diverse, unseen tasks. This suggests that LLMs could be a feasible alternative to human evaluation and other automated metrics in TST evaluation. We compare the results of different LLMs in TST using multiple input prompts. Our findings highlight a strong correlation between (even zero-shot) prompting and human evaluation, showing that LLMs often outperform traditional automated metrics. Furthermore, we introduce the concept of prompt ensembling, demonstrating its ability to enhance the robustness of TST evaluation. This research contributes to the ongoing evaluation of LLMs in diverse tasks, offering insights into successful outcomes and areas of limitation.