# Dynamic Evaluation Framework for Personalized and Trustworthy Agents: A Multi-Session Approach to Preference Adaptability

arXiv: https://arxiv.org/abs/2504.06277 · HTML: https://mlsanity.com/paper/2504.06277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 10 of 10)

- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [On the steerability of large language models toward data-driven personas](https://mlsanity.com/paper/2311.04978/) (2023)
- [Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement](https://mlsanity.com/paper/2402.11060/) (2024)
- [Faithful Persona-based Conversational Dataset Generation with Large Language Models](https://mlsanity.com/paper/2312.10007/) (2023)
- [Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion](https://mlsanity.com/paper/2311.06318/) (2023)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [Ada-Retrieval: An Adaptive Multi-Round Retrieval Paradigm for Sequential Recommendations](https://mlsanity.com/paper/2401.06633/) (2024)
- [Dynamic Information Retrieval: Theoretical Framework and Application](https://mlsanity.com/paper/1601.04605/) (2016)
- [Efficient and Responsible Adaptation of Large Language Models for Robust and Equitable Top-k Recommendations](https://mlsanity.com/paper/2501.04762/) (2025)

## Abstract (arXiv)

Recent advancements in generative AI have significantly increased interest in personalized agents. With increased personalization, there is also a greater need for being able to trust decision-making and action taking capabilities of these agents. However, the evaluation methods for these agents remain outdated and inadequate, often failing to capture the dynamic and evolving nature of user interactions. In this conceptual article, we argue for a paradigm shift in evaluating personalized and adaptive agents. We propose a comprehensive novel framework that models user personas with unique attributes and preferences. In this framework, agents interact with these simulated users through structured interviews to gather their preferences and offer customized recommendations. These recommendations are then assessed dynamically using simulations driven by Large Language Models (LLMs), enabling an adaptive and iterative evaluation process. Our flexible framework is designed to support a variety of agents and applications, ensuring a comprehensive and versatile evaluation of recommendation strategies that focus on proactive, personalized, and trustworthy aspects.