mlsanityData to 7 Oct 2026

PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants

arXiv 2506.09902 ↗First posted 11 Jun 2025cs.CL cs.AI cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have advanced conversational AI assistants. However, systematically evaluating how well these assistants apply personalization--adapting to individual user preferences while completing tasks--remains challenging. Existing personalization benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, failing to capture the complexities of personalized task-oriented assistance. To address this, we introduce PersonaLens, a comprehensive benchmark for evaluating personalization in task-oriented AI assistants. Our benchmark features diverse user profiles equipped with rich preferences and interaction histories, along with two specialized LLM-based agents: a user agent that engages in realistic task-oriented dialogues with AI assistants, and a judge agent that employs the LLM-as-a-Judge paradigm to assess personalization, response quality, and task success. Through extensive experiments with current LLM assistants across diverse tasks, we reveal significant variability in their personalization capabilities, providing crucial insights for advancing conversational AI systems.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

37 of 59 references.

References · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  4. 2023Large Language Modelscited by 774
  5. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  6. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  7. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  8. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 336
  9. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  10. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  11. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  12. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
25 more
  1. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
  2. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  3. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 91
  4. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
  5. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 81
  6. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  7. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  8. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  9. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  10. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 45
  11. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  12. 2023Learning to Rewrite Prompts for Personalized Text Generationcited by 40
  13. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systemscited by 38
  14. 2023Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue Systemcited by 32
  15. 2017Personalization in Goal-Oriented Dialogcited by 28
  16. 2024Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Searchcited by 24
  17. 2024DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialoguescited by 23
  18. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  19. 2023MG-ShopDial: A Multi-Goal Conversational Dataset for e-Commercecited by 19
  20. 2022Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulatorcited by 15
  21. 2024Selective Prompting Tuning for Personalized Conversations with LLMscited by 15
  22. 2022BotsTalk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasetscited by 13
  23. 2024Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systemscited by 11
  24. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  25. 2023Reference-less Analysis of Context Specificity in Translation with Personalised Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 40.

44 papers.

Sort:

By area:

  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  2. 2026Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialoguescited by 16
  3. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 15
  4. 2025Towards Personalized Deep Research: Benchmarks and Evaluationscited by 14
  5. 2026HorizonBench: Long-Horizon Personalization with Evolving Preferencescited by 14
  6. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 13
  7. 2026LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluationcited by 11
  8. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  9. 2026AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignmentcited by 6
  10. 2025PREF: Reference-Free Evaluation of Personalised Text Generation in LLMscited by 5
  11. 2026MemAudit: Auditing Long-Term Agent Memory via Hidden User-State Recoverycited by 5
  12. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
32 more
  1. 2025Do LLMs Recognize Your Latent Preferences? A Benchmark for Latent Information Discovery in Personalized Interactioncited by 3
  2. 2026EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal Worldcited by 3
  3. 2026Rethinking Personalization in Large Language Models at the Token Levelcited by 3
  4. 2026Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimizationcited by 2
  5. 2026Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commercecited by 1
  6. 2026SovereignNegotiation-Bench: Evaluating User-Owned Personal Agents In Delegated Bargaining Under Privacy, Consent, Evidence, And Institutional Pressurecited by 1
  7. 2026From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agentscited by 1
  8. 2026VoiceMem: Streaming Dual-Brain Memory for Real-Time Interactioncited by 1
  9. 2026Hypotheses-Guided Self Distillation for Continual Personalizationcited by 1
  10. 2026After Talking with 1,000 Personas: Learning Preference-Aligned Proactive Assistants From Large-Scale Persona Interactionscited by 0
  11. 2026From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMscited by 0
  12. 2026Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalizationcited by 0
  13. 2026Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systemscited by 0
  14. 2026From Chat to Interview: Agentic Requirements Elicitation with an Experience Ontologycited by 0
  15. 2026Re-Centering Humans in LLM Personalizationcited by 0
  16. 2026Beyond Individual Personas: Aligning Synthetic Dialogue to Population-Level Behavior Distributionscited by 0
  17. 2026X+Slides: Benchmarking Audience-Conditioned Slide Generationcited by 0
  18. 2026Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agentscited by 0
  19. 2026SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Contextcited by 0
  20. 2026IntentTune: Using user demand and personalization to resolve "unknown" query intents for e-commerce searchcited by 0
  21. 2026REDDIT: Forgetting-Resistant Correction of Timestamp Drift in ASR via Replay-Based Distribution Editingcited by 0
  22. 2026Benchmarking the Personalization Capabilities of Large Language Modelscited by 0
  23. 2026LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logscited by 0
  24. 2026Muscle Memory for Agents: Compile not Merely Retrievecited by 0
  25. 2026UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMscited by 0
  26. 2026PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Taskscited by 0
  27. 2026PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistantscited by 0
  28. 2026Evaluating the Hidden Costs of Personalization in Large Language Modelscited by 0
  29. 2026VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferencescited by 0
  30. 2026Inferring Hidden User Models from the Behavior of Personalized LLM Agentscited by 0
  31. 2026One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extractioncited by 0
  32. 2026COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents
  2. 2024CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants
  3. 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
  4. 2025Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction
  5. 2026APeB: Benchmarking Personalization Ability of Large Language Model Agents
  6. 2025ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation
  7. 2025Human vs. Agent in Task-Oriented Conversations
  8. 2026PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users