Dynamic Evaluation Framework for Personalized and Trustworthy Agents: A Multi-Session Approach to Preference Adaptability
On the map: LLM agents and tool use → Agentic recommender systems
Abstract (arXiv)
Recent advancements in generative AI have significantly increased interest in personalized agents. With increased personalization, there is also a greater need for being able to trust decision-making and action taking capabilities of these agents. However, the evaluation methods for these agents remain outdated and inadequate, often failing to capture the dynamic and evolving nature of user interactions. In this conceptual article, we argue for a paradigm shift in evaluating personalized and adaptive agents. We propose a comprehensive novel framework that models user personas with unique attributes and preferences. In this framework, agents interact with these simulated users through structured interviews to gather their preferences and offer customized recommendations. These recommendations are then assessed dynamically using simulations driven by Large Language Models (LLMs), enabling an adaptive and iterative evaluation process. Our flexible framework is designed to support a variety of agents and applications, ensuring a comprehensive and versatile evaluation of recommendation strategies that focus on proactive, personalized, and trustworthy aspects.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use. Builds on 10 of its 33 references that are in the map. First posted on arXiv 8 Mar 2025.
Builds on
10 of 33 references.
References · 10
- 2023LaMP: When Large Language Models Meet Personalizationcited by 425
- 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
- 2023On the steerability of large language models toward data-driven personascited by 53
- 2024Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinementcited by 47
- 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 45
- 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
- 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
- 2024Ada-Retrieval: An Adaptive Multi-Round Retrieval Paradigm for Sequential Recommendationscited by 5
- 2016Dynamic Information Retrieval: Theoretical Framework and Applicationcited by 2
- 2025Efficient and Responsible Adaptation of Large Language Models for Robust and Equitable Top-k Recommendationscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Beyond Static Evaluation: Rethinking the Assessment of Personalized Agent Adaptability in Information Retrieval
- 2023On Generative Agents in Recommendation
- 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
- 2026Learning Personalized Agents from Human Feedback
- 2025A Survey of Personalization: From RAG to Agent
- 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
- 2026From Hidden Profiles to Governable Personalization: Recommender Systems in the Age of LLM Agents
- 2026Personalized Recommendation Tool Learning via Autonomous Language Agents