mlsanityData to 14 Oct 2026

Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agents

arXiv 2610.19349 ↗First posted 14 Oct 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group Relative Policy Optimization (GRPO) provides no reward-relative learning signal when all rollouts receive identical rewards. In long-horizon tasks that require multiple interactions, rewards are given only upon successful task completion, making all-fail groups common early in training and limiting opportunities to practice later steps. Guide Assisted GRPO replays a successful reference prefix before the learner continues. We study how initial positions, handoff units, and movement rules affect unassisted task completion. Across three model sizes on BFCL, starting near completion is more effective under a fixed turn-level threshold rule. Finer handoffs alone do not consistently help, but outcome-informed tool-call selection outperforms GRPO on trained and untrained task categories. Guided starts also improve AppWorld success on both evaluation splits. These findings highlight where learning starts and how starting positions move as key curriculum choices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 22 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

12 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb

Other references · 9

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  5. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  6. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  7. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 27
  8. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 19
  9. 2026Agent-G2: Gaussian Guidance for Agentic Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
  2. 2026VeriGate: Verifier-Gated Step-Level Supervision for GRPO
  3. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  4. 2026Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPO
  5. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  6. 2026Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL
  7. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  8. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?