# ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation

arXiv: https://arxiv.org/abs/2501.14956 · HTML: https://mlsanity.com/paper/2501.14956/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 61 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordination](https://mlsanity.com/paper/2309.11696/) (2023)
- [Learning to Rewrite Prompts for Personalized Text Generation](https://mlsanity.com/paper/2310.00152/) (2023)
- [Learning Personalized Alignment for Evaluating Open-ended Text Generation](https://mlsanity.com/paper/2310.03304/) (2023)

## Built on by (most cited first, 26 of 26)

- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation](https://mlsanity.com/paper/2512.06690/) (2025)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [A Cognitive Writing Perspective for Constrained Long-Form Text Generation](https://mlsanity.com/paper/2502.12568/) (2025)
- [PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI](https://mlsanity.com/paper/2512.24848/) (2025)
- [SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models](https://mlsanity.com/paper/2506.04180/) (2025)
- [Learning from Natural Language Feedback for Personalized Question Answering](https://mlsanity.com/paper/2508.10695/) (2025)
- [Evaluating Style-Personalized Text Generation: Challenges and Directions](https://mlsanity.com/paper/2508.06374/) (2025)
- [Rethinking Personalization in Large Language Models at the Token Level](https://mlsanity.com/paper/2603.06595/) (2026)
- [LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles](https://mlsanity.com/paper/2506.06561/) (2025)
- [Investigating LLM Variability in Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2510.03795/) (2025)
- [IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering](https://mlsanity.com/paper/2510.23536/) (2025)
- [Preference-Aware Rubric Learning for Personalized Evaluation](https://mlsanity.com/paper/2605.31545/) (2026)
- [Iterative Critique-Refine Framework for Enhancing LLM Personalization](https://mlsanity.com/paper/2510.24469/) (2025)
- [Improving User Privacy in Personalized Generation: Client-Side Retrieval-Augmented Modification of Server-Side Generated Speculations](https://mlsanity.com/paper/2601.17569/) (2026)
- [StoryLens: Preference-Aligned Story Rewriting via Context-Aware Narrative Enrichment](https://mlsanity.com/paper/2605.28073/) (2026)
- [PersonalBench: Measuring the Authorship Gap in LLM Personalization](https://mlsanity.com/paper/2608.19746/) (2026)
- [Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.07794/) (2025)
- [Comparative Personalization for Multi-document Summarization](https://mlsanity.com/paper/2509.21562/) (2025)
- [PerQ: Efficient Evaluation of Multilingual Text Personalization Quality](https://mlsanity.com/paper/2509.25903/) (2025)
- [Evaluation of Multilingual LLMs Personalized Text Generation Capabilities Targeting Groups and Social-Media Platforms](https://mlsanity.com/paper/2601.03752/) (2026)
- [A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs](https://mlsanity.com/paper/2608.26177/) (2026)
- [Less Is Personal: Learning Minimal Sufficient User Profiles for Personalized Language Models](https://mlsanity.com/paper/2609.08180/) (2026)
- [One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction](https://mlsanity.com/paper/2609.21626/) (2026)

## Abstract (arXiv)

Evaluating personalized text generated by large language models (LLMs) is challenging, as only the LLM user, i.e., prompt author, can reliably assess the output, but re-engaging the same individuals across studies is infeasible. This paper addresses the challenge of evaluating personalized text generation by introducing ExPerT, an explainable reference-based evaluation framework. ExPerT leverages an LLM to extract atomic aspects and their evidence from the generated and reference texts, match the aspects, and evaluate their alignment based on content and writing style -- two key attributes in personalized text generation. Additionally, ExPerT generates detailed, fine-grained explanations for every step of the evaluation process, enhancing transparency and interpretability. Our experiments demonstrate that ExPerT achieves a 7.2% relative improvement in alignment with human judgments compared to the state-of-the-art text generation evaluation methods. Furthermore, human evaluators rated the usability of ExPerT's explanations at 4.7 out of 5, highlighting its effectiveness in making evaluation decisions more interpretable.