mlsanityData to 14 Oct 2026

Matched-Context Group Policy Optimization for LLM Agent Training

arXiv 2610.13755 ↗First posted 9 Oct 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 47 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

45 of 47 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning20 Aug
  2. 2026MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents20 Aug
  3. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training19 Aug
  4. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents17 Aug
  5. 2026STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training6 Jul
  6. 2026Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning5 Jul
  7. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents24 Jun
  8. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning22 Jun
  9. 2026When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training4 Jun
  10. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning26 May
  11. 2026Milestone-Guided Policy Learning for Long-Horizon Language Agents7 May
  12. 2026RTMC: Step-Level Credit Assignment via Rollout Trees13 Apr
9 more
  1. 2026Hindsight Credit Assignment for Long-Horizon LLM Agents7 Mar
  2. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents1 Mar
  3. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  4. 2025SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph22 Oct
  5. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  6. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  8. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  10. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  12. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
12 more
  1. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  2. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  3. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  5. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  6. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  7. 2026Step-wise Rubric Rewards for LLM Reasoningcited by 14
  8. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  9. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal Statescited by 5
  10. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 5
  11. 2026Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learningcited by 4
  12. 2026Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPOcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
  2. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities
  3. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
  4. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  5. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  6. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  7. 2026RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents
  8. 2026Verified Critical Step Optimization for LLM Agents