# Matched-Context Group Policy Optimization for LLM Agent Training

arXiv: https://arxiv.org/abs/2610.13755 · HTML: https://mlsanity.com/paper/2610.13755/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 45 of its 47 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 45)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents](https://mlsanity.com/paper/2408.07199/) (2024)
- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [VinePPO: Refining Credit Assignment in RL Training of LLMs](https://mlsanity.com/paper/2410.01679/) (2024)
- [Agent Lightning: Train ANY AI Agents with Reinforcement Learning](https://mlsanity.com/paper/2508.03680/) (2025)
- [Reinforcement Learning for Long-Horizon Interactive LLM Agents](https://mlsanity.com/paper/2502.01600/) (2025)
- [Contrastive Preference Learning: Learning from Human Feedback without RL](https://mlsanity.com/paper/2310.13639/) (2023)
- [REBEL: Reinforcement Learning via Regressing Relative Rewards](https://mlsanity.com/paper/2404.16767/) (2024)
- [Hindsight Credit Assignment for Long-Horizon LLM Agents](https://mlsanity.com/paper/2603.08754/) (2026)
- [Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning](https://mlsanity.com/paper/2508.16949/) (2025)
- [Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2602.22817/) (2026)
- [Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.26684/) (2026)
- [SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph](https://mlsanity.com/paper/2510.20022/) (2025)
- [Milestone-Guided Policy Learning for Long-Horizon Language Agents](https://mlsanity.com/paper/2605.06078/) (2026)
- [Step-wise Rubric Rewards for LLM Reasoning](https://mlsanity.com/paper/2605.17291/) (2026)
- [Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.22995/) (2026)

## Abstract (arXiv)

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.