mlsanityData to 7 Oct 2026

Training Proactive and Personalized LLM Agents

arXiv 2511.02208 ↗First posted 4 Nov 2025cs.AI cs.CL cs.LGcited by 42

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Despite rapid progress, current AI agents are primarily optimized for isolated task completion. We argue for a paradigm shift toward training agents as collaborators that communicate and adapt to people. To facilitate this shift in real-world complex applications, we first formalize three dimensions of collaborative AI agents: Productivity, Proactivity, and Personalization (PPP). We introduce UserVille, an interactive environment with configurable LLM-based user simulators and user-centric feedback to evaluate these dimensions, and propose a multi-objective reinforcement learning framework that optimizes them using rewards from task outcomes, question effort, and preference adherence. On two real-world agentic tasks (SWE-Bench and BrowseComp-Plus), PPP-trained agents outperform strong LLM baselines (including GPT-5) by an average of 16.7 points, ask more targeted questions, and generalize to unseen preferences and tasks. A follow-up user study further highlights the importance of user-centric feedback for training collaborative agents that are both more effective and easier to supervise.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 31 of its 43 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

31 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025UserBench: An Interactive Gym Environment for User-Centric Agents29 Jul
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  8. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  10. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  11. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  12. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
17 more
  1. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  2. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  3. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  4. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  5. 2025LocAgent: Graph-Guided LLM Agents for Code Localizationcited by 82
  6. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  7. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 37
  8. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  9. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  10. 2025Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflowscited by 27
  11. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  12. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  13. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systemscited by 18
  14. 2023Answering Ambiguous Questions via Iterative Promptingcited by 13
  15. 2025TOM-SWE: User Mental Modeling For Software Engineering Agentscited by 11
  16. 2024Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Informationcited by 5
  17. 2025Completion neq Collaboration: Scaling Collaborative Effort with Agentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 40.

42 papers.

Sort:

By area:

  1. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  2. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 33
  3. 2026ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Datacited by 15
  4. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 15
  5. 2026Value of Information: A Framework for Human-Agent Communicationcited by 10
  6. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  7. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  8. 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Supportcited by 9
  9. 2026Learning to Simulate Human Dialoguecited by 9
  10. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  11. 2026Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generationcited by 7
  12. 2026Learning Next Action Predictors from Human-Computer Interactioncited by 7
30 more
  1. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  2. 2026AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraintscited by 6
  3. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  4. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 4
  5. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  6. 2026TriggerBench: Investigating Prospective Memory for Large Language Modelscited by 4
  7. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 3
  8. 2026Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interactioncited by 3
  9. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 3
  10. 2026LLMs Get Lost in Evolving User Intentcited by 3
  11. 2026Humans are Missing from AI Coding Agent Researchcited by 3
  12. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  13. 2026PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agentcited by 2
  14. 2026Agentic Coding Needs Proactivity, Not Just Autonomycited by 2
  15. 2026Simulating Human Memory with Language Modelscited by 2
  16. 2026Do Proactive Agents Really Need an LLM to Decide When to Wake and What to Anchor?cited by 2
  17. 2026Steering LLMs via Scalable Interactive Oversightcited by 1
  18. 2026Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mindcited by 1
  19. 2026Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestrationcited by 1
  20. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 1
  21. 2026Decision-Centric Design for LLM Systemscited by 0
  22. 2026TIDE: Proactive Multi-Problem Discovery via Template-Guided Iterationcited by 0
  23. 2026Communication Policy Evolution for Proactive LLM Agentscited by 0
  24. 2026SkillSmith: Learning to Compose Parametric Skills and Textual Knowledgecited by 0
  25. 2026Intent-Driven Situation Tracking for User-Centric Multi-Turn Agentscited by 0
  26. 2026PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Taskscited by 0
  27. 2026AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamicscited by 0
  28. 2026From Migration to Calibration: Preserving Agent Capabilities across Models, Jurisdictions, and Scalecited by 0
  29. 2026Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agentscited by 0
  30. 2026Beyond Mode Collapse: Generating Diverse Synthetic Expert Conversations via Generative Flow Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving LLM Collaboration via Multi-Agent Preference Learning
  2. 2025PersonaAgent: Bridging Memory and Action for Personalized LLM Agents
  3. 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
  4. 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
  5. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agents
  6. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory
  7. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  8. 2026UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training