# Multidimensional Evaluation for Text Style Transfer Using ChatGPT

arXiv: https://arxiv.org/abs/2304.13462 · HTML: https://mlsanity.com/paper/2304.13462/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://mlsanity.com/paper/1808.04339/) (2018)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1905.10060/) (2019)
- [ChatGPT: A Meta-Analysis after 2.5 Months](https://mlsanity.com/paper/2302.13795/) (2023)
- [ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots](https://mlsanity.com/paper/2302.06466/) (2023)
- [Parallel Data Augmentation for Formality Style Transfer](https://mlsanity.com/paper/2005.07522/) (2020)
- ["I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data](https://mlsanity.com/paper/2212.05856/) (2022)
- [Multi-Task Neural Models for Translating Between Styles Within and Across Languages](https://mlsanity.com/paper/1806.04357/) (2018)
- [Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer](https://mlsanity.com/paper/2105.06947/) (2021)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric](https://mlsanity.com/paper/2004.05001/) (2020)
- [Stable Style Transformer: Delete and Generate Approach with Encoder-Decoder for Text Style Transfer](https://mlsanity.com/paper/2005.12086/) (2020)
- [A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer](https://mlsanity.com/paper/1906.01833/) (2019)
- [Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information Maximization](https://mlsanity.com/paper/2010.05090/) (2020)
- [Reinforced Rewards Framework for Text Style Transfer](https://mlsanity.com/paper/2005.05256/) (2020)

## Built on by (most cited first, 12 of 12)

- [CAT-LLM: Style-enhanced Large Language Models with Text Style Definition for Chinese Article-style Transfer](https://mlsanity.com/paper/2401.05707/) (2024)
- [Learning Evaluation Models from Large Language Models for Sequence Generation](https://mlsanity.com/paper/2308.04386/) (2023)
- [VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization](https://mlsanity.com/paper/2311.00807/) (2023)
- [LMStyle Benchmark: Evaluating Text Style Transfer for Chatbots](https://mlsanity.com/paper/2403.08943/) (2024)
- [Evaluating Style-Personalized Text Generation: Challenges and Directions](https://mlsanity.com/paper/2508.06374/) (2025)
- [T5Score: A Methodology for Automatically Assessing the Quality of LLM Generated Multi-Document Topic Sets](https://mlsanity.com/paper/2407.17390/) (2024)
- [Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts](https://mlsanity.com/paper/2408.09688/) (2024)
- [Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics](https://mlsanity.com/paper/2502.15022/) (2025)
- [PSST: A Benchmark for Evaluation-driven Text Public-Speaking Style Transfer](https://mlsanity.com/paper/2311.08389/) (2023)
- [Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?](https://mlsanity.com/paper/2408.04029/) (2024)
- [FiSTECH: Financial Style Transfer to Enhance Creativity without Hallucinations in LLMs](https://mlsanity.com/paper/2408.05365/) (2024)
- [Translation as a Decision Space: A Multi-Agent Perspective on Low-Resource Dialect Generation](https://mlsanity.com/paper/2609.04048/) (2026)

## Abstract (arXiv)

We investigate the potential of ChatGPT as a multidimensional evaluator for the task of Text Style Transfer, alongside, and in comparison to, existing automatic metrics as well as human judgements. We focus on a zero-shot setting, i.e. prompting ChatGPT with specific task instructions, and test its performance on three commonly-used dimensions of text style transfer evaluation: style strength, content preservation, and fluency. We perform a comprehensive correlation analysis for two transfer directions (and overall) at different levels. Compared to existing automatic metrics, ChatGPT achieves competitive correlations with human judgments. These preliminary results are expected to provide a first glimpse into the role of large language models in the multidimensional evaluation of stylized text generation.