# PersoBench: Benchmarking Personalized Response Generation in Large Language Models

arXiv: https://arxiv.org/abs/2410.03198 · HTML: https://mlsanity.com/paper/2410.03198/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [Personalizing Dialogue Agents via Meta-Learning](https://mlsanity.com/paper/1905.10033/) (2019)
- [FELM: Benchmarking Factuality Evaluation of Large Language Models](https://mlsanity.com/paper/2310.00741/) (2023)
- [TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models](https://mlsanity.com/paper/2306.11507/) (2023)
- [Prompt Design and Engineering: Introduction and Advanced Methods](https://mlsanity.com/paper/2401.14423/) (2024)
- [PAL: Persona-Augmented Emotional Support Conversation Generation](https://mlsanity.com/paper/2212.09235/) (2022)
- [DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems](https://mlsanity.com/paper/2407.10701/) (2024)
- [LLMs + Persona-Plug = Personalized LLMs](https://mlsanity.com/paper/2409.11901/) (2024)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models](https://mlsanity.com/paper/2405.18027/) (2024)
- [Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2405.17974/) (2024)
- [Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation](https://mlsanity.com/paper/2405.18346/) (2024)
- [A Personalized Dialogue Generator with Implicit User Persona Detection](https://mlsanity.com/paper/2204.07372/) (2022)
- [Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialogues](https://mlsanity.com/paper/2301.04871/) (2023)
- [Towards Robust Personalized Dialogue Generation via Order-Insensitive Representation Regularization](https://mlsanity.com/paper/2305.12782/) (2023)
- [Harnessing Large Language Models' Empathetic Response Generation Capabilities for Online Mental Health Counselling Support](https://mlsanity.com/paper/2310.08017/) (2023)
- [Health-LLM: Personalized Retrieval-Augmented Disease Prediction System](https://mlsanity.com/paper/2402.00746/) (2024)

## Built on by (most cited first, 13 of 13)

- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations](https://mlsanity.com/paper/2505.14106/) (2025)
- [TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation](https://mlsanity.com/paper/2510.25536/) (2025)
- [HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation](https://mlsanity.com/paper/2505.00038/) (2025)
- [PrefDisco: Benchmarking Proactive Personalized Reasoning](https://mlsanity.com/paper/2510.00177/) (2025)
- [PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systems](https://mlsanity.com/paper/2602.00016/) (2026)
- [Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2505.21907/) (2025)
- [Rethinking Personalization in Large Language Models at the Token Level](https://mlsanity.com/paper/2603.06595/) (2026)
- [GOD model: Privacy Preserved AI School for Personal Assistant](https://mlsanity.com/paper/2502.18527/) (2025)
- [PersoPilot: An Adaptive AI-Copilot for Transparent Contextualized Persona Classification and Personalized Response Generation](https://mlsanity.com/paper/2602.04540/) (2026)
- [PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluation](https://mlsanity.com/paper/2602.04493/) (2026)
- [Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs](https://mlsanity.com/paper/2606.09038/) (2026)
- [Benchmarking the Personalization Capabilities of Large Language Models](https://mlsanity.com/paper/2607.20471/) (2026)

## Abstract (arXiv)

While large language models (LLMs) have exhibited impressive conversational capabilities, their proficiency in delivering personalized responses remains unclear. Although recent benchmarks automatically evaluate persona consistency in role-playing contexts using LLM-based judgment, the evaluation of personalization in response generation remains underexplored. To address this gap, we present an automated benchmarking pipeline, PersoBench, to evaluate the personalization ability of LLMs in persona-aware dialogue generation within a zero-shot setting. Our framework employs a structured pipeline comprising speaker-aware annotation, task-specific and context-driven prompt construction, response post-processing, and automated evaluation across multiple dimensions of generation quality. In particular, the pipeline performs text preprocessing and speaker labeling, constructs structured prompts with task instructions and LLM roles, validates response format, and evaluates valid outputs across fluency, personalization, diversity, and coherence. We assess the performance of four open-source and four closed-source LLMs using well-known datasets and a range of explicit metrics. Our findings reveal that while LLMs excel at generating fluent and diverse responses, they are far from satisfactory in delivering personalized and coherent responses, considering both the conversation context and the provided personas.