mlsanityData to 14 Oct 2026

PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents

arXiv 2610.14636 ↗First posted 10 Oct 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Terminal rewards make multi-turn agents trainable without step-level annotations, but group-relative advantages provide little guidance about which intermediate decisions to adjust. Failed trajectories also contain reasonable actions, while unfavorable decision patterns can recur as training proceeds. We introduce PEARL (Process-Experience Aware Reinforcement Learning), which improves local credit assignment without an additional critic. Given two continuations from the same history with different outcomes, PEARL locates their first divergence and builds a feature representation of each branch's local decision. It clusters these decision features in recent failure and success memories to identify recurring local patterns. Using this pattern memory, PEARL selectively strengthens negative advantages at lower-reward decisions, while leaving positive advantages and the base policy optimizer unchanged. On AppWorld, PEARL improves over ARPO on Normal and Challenge tasks at both 4B and 8B scales. On ALFWorld, it achieves 97.9% mean success at 4B, 2.7 points above the strongest baseline, while remaining competitive at 8B. Ablations of shaping location, success-bank filtering, and memory window support the value of selective, recurring-pattern-based updates.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 29 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

15 of 29 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  5. 2025Group Sequence Policy Optimizationcited by 835
  6. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  7. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  8. 2025Agentic Reinforced Policy Optimizationcited by 172
  9. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  10. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  11. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 25
  12. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
3 more
  1. 2026StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstractioncited by 4
  2. 2026The Past Is Not Past: Memory-Enhanced Dynamic Reward Shapingcited by 1
  3. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Trainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning
  2. 2026SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training
  3. 2026PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use
  4. 2026SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents
  5. 2026CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planning
  6. 2026C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning
  7. 2026ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning
  8. 2026HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning