mlsanityData to 7 Oct 2026

Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Training

arXiv 2602.19225 ↗First posted 22 Feb 2026cs.AIcited by 18

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Multi-turn LLM agents are becoming pivotal to production systems, spanning customer service automation, e-commerce assistance, and interactive task management, where accurately distinguishing high-value informative signals from stochastic noise is critical for sample-efficient training. In real-world scenarios, a failure in a trivial task may reflect random instability, whereas success in a high-difficulty task signifies a genuine capability breakthrough. Yet, existing group-based policy optimization methods rigidly rely on statistical deviation within discrete batches, frequently misallocating credit when task difficulty fluctuates. To address this issue, we propose Proximity-based Multi-turn Optimization (ProxMO), a practical and robust framework engineered specifically for the constraints of real-world deployment. ProxMO integrates global context via two lightweight mechanisms: success-rate-aware modulation dynamically adapts gradient intensity based on episode-level difficulty, while proximity-based soft aggregation derives baselines through continuous semantic weighting at the step level. Extensive evaluations on ALFWorld and WebShop benchmarks demonstrate that ProxMO yields substantial performance gains over existing baselines with negligible computational cost. Ablation studies further validate the independent and synergistic efficacy of both mechanisms. Crucially, ProxMO offers plug-and-play compatibility with standard GRPO frameworks, facilitating immediate, low-friction adoption in existing industrial training pipelines. Our implementation is available at: hrefhttps://anonymous.4open.science/r/proxmo-B7E7/README.mdhttps://anonymous.4open.science/r/proxmo.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 30 references that are in the map. First posted on arXiv 22 Feb 2026.

Builds on

20 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning1 May
  3. 2025ToolRL: Reward is All Tool Learning Needs16 Apr

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
5 more
  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  2. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  3. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  4. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  5. 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026Policy and World Modeling Co-Training for Language Agentscited by 11
  2. 2026The Interplay of Harness Design and Post-Training in LLM Agentscited by 8
  3. 2026Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibrationcited by 7
  4. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7
  5. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  6. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 6
  7. 2026Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectoriescited by 4
  8. 2026Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderationcited by 3
  9. 2026From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Spacecited by 3
  10. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generationcited by 1
  11. 2026IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrievalcited by 1
  12. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capabilitycited by 1
6 more
  1. 2026How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learningcited by 1
  2. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  3. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
  4. 2026MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignmentcited by 0
  5. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learningcited by 0
  6. 2026Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MASkills: Continual Skills Optimization for Multi-Agent LLM Systems
  2. 2026Optimizing Service Operations via LLM-Powered Multi-Agent Simulation
  3. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  4. 2026IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
  5. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  6. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  7. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?
  8. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents