# Training Proactive and Personalized LLM Agents

arXiv: https://arxiv.org/abs/2511.02208 · HTML: https://mlsanity.com/paper/2511.02208/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [Scaling Long-Horizon LLM Agent via Context-Folding](https://mlsanity.com/paper/2510.11967/) (2025)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks](https://mlsanity.com/paper/2503.15478/) (2025)
- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)
- [UserBench: An Interactive Gym Environment for User-Centric Agents](https://mlsanity.com/paper/2507.22034/) (2025)
- [OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety](https://mlsanity.com/paper/2507.06134/) (2025)
- [ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification](https://mlsanity.com/paper/2310.10996/) (2023)
- [Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering](https://mlsanity.com/paper/2502.13069/) (2025)
- [Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction](https://mlsanity.com/paper/2303.09224/) (2023)
- [UserRL: Training Interactive User-Centric Agent via Reinforcement Learning](https://mlsanity.com/paper/2509.19736/) (2025)
- [Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows](https://mlsanity.com/paper/2507.08149/) (2025)
- [ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning](https://mlsanity.com/paper/2502.14860/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [Answering Ambiguous Questions via Iterative Prompting](https://mlsanity.com/paper/2307.03897/) (2023)
- [TOM-SWE: User Mental Modeling For Software Engineering Agents](https://mlsanity.com/paper/2510.21903/) (2025)
- [Completion neq Collaboration: Scaling Collaborative Effort with Agents](https://mlsanity.com/paper/2510.25744/) (2025)

## Built on by (most cited first, 30 of 46)

- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems](https://mlsanity.com/paper/2604.14228/) (2026)
- [Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents](https://mlsanity.com/paper/2603.26233/) (2026)
- [ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data](https://mlsanity.com/paper/2602.04482/) (2026)
- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning](https://mlsanity.com/paper/2602.21492/) (2026)
- [InfoPO: Information-Driven Policy Optimization for User-Centric Agents](https://mlsanity.com/paper/2603.00656/) (2026)
- [MindEval: Benchmarking Language Models on Multi-turn Mental Health Support](https://mlsanity.com/paper/2511.18491/) (2025)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [LLMs Get Lost in Evolving User Intent](https://mlsanity.com/paper/2607.20734/) (2026)
- [Learning to Simulate Human Dialogue](https://mlsanity.com/paper/2601.04436/) (2026)
- [Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation](https://mlsanity.com/paper/2602.03045/) (2026)
- [Learning Next Action Predictors from Human-Computer Interaction](https://mlsanity.com/paper/2603.05923/) (2026)
- [ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions](https://mlsanity.com/paper/2605.20087/) (2026)
- [AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints](https://mlsanity.com/paper/2606.05622/) (2026)
- [DiscoverLLM: From Executing Intents to Discovering Them](https://mlsanity.com/paper/2602.03429/) (2026)
- [One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL](https://mlsanity.com/paper/2608.12253/) (2026)
- [Learning Steerable Clarification Policies with Collaborative Self-play](https://mlsanity.com/paper/2512.04068/) (2025)
- [Bridging the Agent-World Gap: Text World Models for LLM-based Agents](https://mlsanity.com/paper/2606.09032/) (2026)
- [TriggerBench: Investigating Prospective Memory for Large Language Models](https://mlsanity.com/paper/2606.23459/) (2026)
- [Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction](https://mlsanity.com/paper/2603.23550/) (2026)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agent](https://mlsanity.com/paper/2604.07645/) (2026)
- [Agentic Coding Needs Proactivity, Not Just Autonomy](https://mlsanity.com/paper/2605.06717/) (2026)
- [Humans are Missing from AI Coding Agent Research](https://mlsanity.com/paper/2608.12355/) (2026)
- [Steering LLMs via Scalable Interactive Oversight](https://mlsanity.com/paper/2602.04210/) (2026)
- [Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization](https://mlsanity.com/paper/2602.11351/) (2026)
- [Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind](https://mlsanity.com/paper/2602.13832/) (2026)
- [Simulating Human Memory with Language Models](https://mlsanity.com/paper/2605.25680/) (2026)

## Abstract (arXiv)

Despite rapid progress, current AI agents are primarily optimized for isolated task completion. We argue for a paradigm shift toward training agents as collaborators that communicate and adapt to people. To facilitate this shift in real-world complex applications, we first formalize three dimensions of collaborative AI agents: Productivity, Proactivity, and Personalization (PPP). We introduce UserVille, an interactive environment with configurable LLM-based user simulators and user-centric feedback to evaluate these dimensions, and propose a multi-objective reinforcement learning framework that optimizes them using rewards from task outcomes, question effort, and preference adherence. On two real-world agentic tasks (SWE-Bench and BrowseComp-Plus), PPP-trained agents outperform strong LLM baselines (including GPT-5) by an average of 16.7 points, ask more targeted questions, and generalize to unseen preferences and tasks. A follow-up user study further highlights the importance of user-centric feedback for training collaborative agents that are both more effective and easier to supervise.