mlsanityData to 14 Oct 2026

From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning

arXiv 2605.23382 ↗First posted 22 May 2026cs.CLcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Agentic reinforcement learning (Agentic RL) has achieved strong progress in tasks with clear success signals. However, many real-world agent applications require user-conditioned behavior: the same query may call for different planning strategies and tool-use decisions across users. This setting raises key challenges: generic rewards cannot capture heterogeneous user preferences, observed behaviors are entangled with conformity effects, and flat memories cannot support personalized skill retrieval. To this end, we propose a unified personalized Agentic RL framework that embeds personalization into training-time optimization. At its core is Personalized Anchor Reward-Decoupled Policy Optimization (PARPO), which decouples generic task-quality rewards from personalized preference rewards and uses user-specific anchors to stabilize learning under heterogeneous reward scales. We further introduce a two-stage preference-disentangled reward model and Preference-Aligned Skill Evolution Graph Memory (PSGM) for personalized supervision and preference-aligned skill retrieval. Together, they form a closed loop of preference identification, policy optimization, and structured skill accumulation. Experiments on ETAPP, ETAPP-Hard, and SJAgent show that our framework consistently outperforms strong memory and RL baselines. Code and data are included in the supplementary materials.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 44 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

39 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents18 Feb
  2. 2025Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents17 Sep
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  5. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr

Other references · 34

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2025Group Sequence Policy Optimizationcited by 835
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  10. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  12. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
22 more
  1. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  2. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
  3. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  4. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  5. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  6. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  7. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  8. 2026SimpleMem: Efficient Lifelong Memory for LLM Agentscited by 149
  9. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  10. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  11. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  12. 2026MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memorycited by 129
  13. 2024Large Language Models Empowered Personalized Web Agentscited by 66
  14. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agentscited by 32
  15. 2025Preference-Aware Memory Update for Long-Term LLM Agentscited by 20
  16. 2026Learning Personalized Agents from Human Feedbackcited by 18
  17. 2025Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivitycited by 17
  18. 2025DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agentscited by 17
  19. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  20. 2026Beyond Dialogue Time: Temporal Semantic Memory for Personalized LLM Agentscited by 14
  21. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoningcited by 11
  22. 2025SERL: Self-Examining Reinforcement Learning on Open-Domaincited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  2. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
  3. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
  4. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling
  5. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  6. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  7. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning
  8. 2025DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition