mlsanityData to 14 Oct 2026

Multidimensional Evaluation for Text Style Transfer Using ChatGPT

arXiv 2304.13462 ↗First posted 26 Apr 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We investigate the potential of ChatGPT as a multidimensional evaluator for the task of Text Style Transfer, alongside, and in comparison to, existing automatic metrics as well as human judgements. We focus on a zero-shot setting, i.e. prompting ChatGPT with specific task instructions, and test its performance on three commonly-used dimensions of text style transfer evaluation: style strength, content preservation, and fluency. We perform a comprehensive correlation analysis for two transfer directions (and overall) at different levels. Compared to existing automatic metrics, ChatGPT achieves competitive correlations with human judgments. These preliminary results are expected to provide a first glimpse into the role of large language models in the multidimensional evaluation of stylized text generation.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

34 of 44 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  10. 2023Mathematical Capabilities of ChatGPTcited by 268
  11. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  12. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
22 more
  1. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  2. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  3. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  4. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  5. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  6. 2019A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfercited by 65
  7. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 51
  8. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  9. 2020Parallel Data Augmentation for Formality Style Transfercited by 39
  10. 2022"I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Datacited by 32
  11. 2018Multi-Task Neural Models for Translating Between Styles Within and Across Languagescited by 28
  12. 2021Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfercited by 28
  13. 2021Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfercited by 25
  14. 2020Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metriccited by 23
  15. 2020Stable Style Transformer: Delete and Generate Approach with Encoder-Decoder for Text Style Transfercited by 22
  16. 2019A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfercited by 20
  17. 2020Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information Maximizationcited by 19
  18. 2020Reinforced Rewards Framework for Text Style Transfercited by 12
  19. 2022Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfercited by 12
  20. 2021Generic resources are what you need: Style transfer tasks without task-specific parallel training datacited by 10
  21. 2020Robust Front-End for Multi-Channel ASR using Flow-Based Density Estimationcited by 9
  22. 2022Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfercited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 2; 2026: 1.

12 papers.

  1. 2024CAT-LLM: Style-enhanced Large Language Models with Text Style Definition for Chinese Article-style Transfercited by 10
  2. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  3. 2023VQA-GEN: A Visual Question Answering Benchmark for Domain Generalizationcited by 5
  4. 2024LMStyle Benchmark: Evaluating Text Style Transfer for Chatbotscited by 5
  5. 2025Evaluating Style-Personalized Text Generation: Challenges and Directionscited by 3
  6. 2024T5Score: A Methodology for Automatically Assessing the Quality of LLM Generated Multi-Document Topic Setscited by 2
  7. 2024Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcriptscited by 2
  8. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 2
  9. 2023PSST: A Benchmark for Evaluation-driven Text Public-Speaking Style Transfercited by 0
  10. 2024Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?cited by 0
  11. 2024FiSTECH: Financial Style Transfer to Enhance Creativity without Hallucinations in LLMscited by 0
  12. 2026Translation as a Decision Space: A Multi-Agent Perspective on Low-Resource Dialect Generationcited by 0