# Evaluating Style-Personalized Text Generation: Challenges and Directions

arXiv: https://arxiv.org/abs/2508.06374 · HTML: https://mlsanity.com/paper/2508.06374/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 79 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Builds on (most cited first, 30 of 42)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews](https://mlsanity.com/paper/1303.4402/) (2013)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [Do LLMs write like humans? Variation in grammatical and rhetorical styles](https://mlsanity.com/paper/2410.16107/) (2024)
- [Evaluating Style Transfer for Text](https://mlsanity.com/paper/1904.02295/) (2019)
- [Controllable Text Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2408.12599/) (2024)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [Learning to Rewrite Prompts for Personalized Text Generation](https://mlsanity.com/paper/2310.00152/) (2023)
- [StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples](https://mlsanity.com/paper/2410.12757/) (2024)
- [Personalized Text Generation with Contrastive Activation Steering](https://mlsanity.com/paper/2503.05213/) (2025)
- [ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation](https://mlsanity.com/paper/2501.14956/) (2025)
- [Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild](https://mlsanity.com/paper/2505.16023/) (2025)
- [Semi-supervised Text Style Transfer: Cross Projection in Latent Space](https://mlsanity.com/paper/1909.11493/) (2019)
- [Personalized Text Generation with Fine-Grained Linguistic Control](https://mlsanity.com/paper/2402.04914/) (2024)

## Built on by (most cited first, 3 of 3)

- [RT-SFT: Text Style Transfer from Non-Parallel Corpora by Roundtrip Translation](https://mlsanity.com/paper/2602.15013/) (2026)
- [Interpreting Style Representations via Style-Eliciting Prompts](https://mlsanity.com/paper/2606.05716/) (2026)
- [Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation](https://mlsanity.com/paper/2610.19166/) (2026)

## Abstract (arXiv)

With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing area of interest. However, style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context, which makes it uniquely challenging. Although prior research has introduced benchmarks and metrics for this area, they tend to be non-standardized and have known limitations (e.g., poor correlation with human subjects). LLMs have been found to not capture author-specific style well, it follows that the metrics themselves must be scrutinized carefully. In this work we critically examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges. We evaluate these metrics using our proposed style discrimination benchmark, which spans eight diverse writing tasks across three evaluation settings: domain discrimination, authorship attribution, and LLM-generated personalized vs non-personalized discrimination. We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods, and conclude by providing guidance on how to reliably assess style-personalized text generation.