mlsanityData to 14 Oct 2026

Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale

arXiv 2504.14225 ↗First posted 19 Apr 2025cs.CLcited by 178

Not on the map. Why?

Cited by 62 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks -- from offering writing support to delivering tailored recommendations or consultations. Over time, the interaction history between a user and an LLM can provide extensive information about an individual's traits and preferences. However, open questions remain on how well LLMs today can effectively leverage such history to (1) internalize the user's inherent traits and preferences, (2) track how the user profiling and preferences evolve over time, and (3) generate personalized responses accordingly in new scenarios. In this work, we introduce the PERSONAMEM benchmark. PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories, each containing up to 60 sessions of multi-turn conversations across 15 real-world tasks that require personalization. Given an in-situ user query, i.e. query issued by the user from the first-person perspective, we evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile. We observe that current LLMs still struggle to recognize the dynamic evolution in users' profiles over time through direct prompting approaches. As a consequence, LLMs often fail to deliver responses that align with users' current situations and preferences, with frontier models such as GPT-4.1, o4-mini, GPT-4.5, o1, or Gemini-2.0 achieving only around 50% overall accuracy, suggesting room for improvement. We hope that PERSONAMEM, along with the user profile and conversation simulation pipeline, can facilitate future research in the development of truly user-aware chatbots. Code and data are available at github.com/bowen-upenn/PersonaMem.

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

45 of 62 references.

References · 45

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
33 more
  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  4. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  5. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  6. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  7. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  8. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  9. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  11. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  12. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  13. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 108
  14. 2024Can LLM be a Personalized Judge?cited by 100
  15. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 97
  16. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 92
  17. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  18. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  19. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  20. 2022POTATO: The Portable Text Annotation Toolcited by 64
  21. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
  22. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  23. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 34
  24. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  25. 2025PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Datacited by 27
  26. 2022Preference Dynamics Under Personalized Recommendationscited by 25
  27. 2025Toward Multi-Session Personalized Conversation: A Large-Scale Dataset and Hierarchical Tree Framework for Implicit Reasoningcited by 21
  28. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  29. 2024Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Modelscited by 19
  30. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  31. 2024WildfireGPT: Tailored Large Language Model for Wildfire Analysiscited by 15
  32. 2024PersoBench: Benchmarking Personalized Response Generation in Large Language Modelscited by 13
  33. 2023Automated Evaluation of Personalized Text Generation using Large Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 156.

178 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025MemOS: A Memory OS for AI Systemcited by 138
  3. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 88
  4. 2025HaluMem: Evaluating Hallucinations in Memory Systems of Agentscited by 54
  5. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  6. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 36
  7. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agentscited by 32
  8. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  9. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 24
  10. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  11. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 23
  12. 2026OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agentscited by 22
38 more
  1. 2026Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialoguescited by 22
  2. 2026AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversationscited by 20
  3. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 18
  4. 2026MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalizationcited by 17
  5. 2026HorizonBench: Long-Horizon Personalization with Evolving Preferencescited by 17
  6. 2025Towards Personalized Deep Research: Benchmarks and Evaluationscited by 16
  7. 2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalizationcited by 15
  8. 2025TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulationcited by 15
  9. 2025DeepPersona: A Generative Engine for Scaling Deep Synthetic Personascited by 15
  10. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 14
  11. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  12. 2026HyperMem: Hypergraph Memory for Long-Term Conversationscited by 14
  13. 2025ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planningcited by 13
  14. 2026GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agentscited by 13
  15. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  16. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactionscited by 12
  17. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memorycited by 12
  18. 2026Choosing How to Remember: Adaptive Memory Structures for LLM Agentscited by 11
  19. 2026LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluationcited by 11
  20. 2026AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agentscited by 11
  21. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 11
  22. 2026Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactionscited by 10
  23. 2026AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignmentcited by 10
  24. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  25. 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
  26. 2025RGMem: Renormalization Group-inspired Memory Evolution for Language Agentscited by 7
  27. 2026MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaborationcited by 7
  28. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  29. 2026AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraintscited by 7
  30. 2025Expert Preference-based Evaluation of Automated Related Work Generationcited by 6
  31. 2026CloneMem: Benchmarking Long-Term Memory for AI Clonescited by 6
  32. 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 6
  33. 2026H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactionscited by 6
  34. 2026MemAudit: Auditing Long-Term Agent Memory via Hidden User-State Recoverycited by 6
  35. 2025PREF: Reference-Free Evaluation of Personalised Text Generation in LLMscited by 5
  36. 2025GMTRouter: Personalized LLM Router over Multi-turn User Interactionscited by 5
  37. 2026Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalizationcited by 5
  38. 2026Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalizationcited by 5

50 most cited of 178 All 178 (JSON)