# Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity

arXiv: https://arxiv.org/abs/2503.00771 · HTML: https://mlsanity.com/paper/2503.00771/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 2 Mar 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)
- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 23 of 23)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs](https://mlsanity.com/paper/2502.09597/) (2025)
- [Conversational Health Agents: A Personalized LLM-Powered Agent Framework](https://mlsanity.com/paper/2310.02374/) (2023)
- [Aligning to Thousands of Preferences via System Message Generalization](https://mlsanity.com/paper/2405.17977/) (2024)
- [Large Language Models Empowered Personalized Web Agents](https://mlsanity.com/paper/2410.17236/) (2024)
- [AI PERSONA: Towards Life-long Personalization of LLMs](https://mlsanity.com/paper/2412.13103/) (2024)
- [Guided Profile Generation Improves Personalization with LLMs](https://mlsanity.com/paper/2409.13093/) (2024)

## Built on by (most cited first, 17 of 17)

- [MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations](https://mlsanity.com/paper/2507.21428/) (2025)
- [PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction](https://mlsanity.com/paper/2602.06714/) (2026)
- [Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis](https://mlsanity.com/paper/2603.15483/) (2026)
- [FamilyTool: A Multi-hop Personalized Tool Use Benchmark](https://mlsanity.com/paper/2504.06766/) (2025)
- [Memory-Induced Tool-Drift in LLM Agents](https://mlsanity.com/paper/2605.24941/) (2026)
- [AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation](https://mlsanity.com/paper/2603.03761/) (2026)
- [Structuring the Space of Sociotechnical Alignment](https://mlsanity.com/paper/2607.01250/) (2026)
- [From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents](https://mlsanity.com/paper/2608.02171/) (2026)
- [Building Persona-Based Agents On Demand: Tailoring Multi-Agent Workflows to User Needs](https://mlsanity.com/paper/2604.27882/) (2026)
- [From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.23382/) (2026)
- [Personalized Turn-Level User Conversation Satisfaction Benchmark](https://mlsanity.com/paper/2605.29711/) (2026)
- [Communication Policy Evolution for Proactive LLM Agents](https://mlsanity.com/paper/2606.14314/) (2026)
- [UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs](https://mlsanity.com/paper/2608.10042/) (2026)
- [PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants](https://mlsanity.com/paper/2608.26180/) (2026)
- [Behaviorally Grounded User Profiles from the Wild for Personalized Alignment and Multi-Perspective Reasoning](https://mlsanity.com/paper/2609.00014/) (2026)
- [Inferring Hidden User Models from the Behavior of Personalized LLM Agents](https://mlsanity.com/paper/2609.03815/) (2026)
- [PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents](https://mlsanity.com/paper/2609.34930/) (2026)

## Abstract (arXiv)

Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. However, most of the current benchmarks primarily focus on either personalization of text generation or direct tool-utilizing, without considering both. In this work, we introduce a novel benchmark ETAPP for evaluating personalized tool invocation, establishing a sandbox environment, and a comprehensive dataset of 800 testing cases covering diverse user profiles. To improve the accuracy of our evaluation, we propose a key-point-based LLM evaluation method, mitigating biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference. Additionally, we evaluate the excellent LLMs and provide an in-depth analysis. Furthermore, we investigate the impact of different tool-invoking strategies on LLMs' personalization performance and the effects of fine-tuning in our task. The effectiveness of our preference-setting and key-point-based evaluation method is also validated. Our findings offer insights into improving personalized LLM agents. Our Code is available at https://github.com/hypasd-art/ETAPP.