mlsanityData to 14 Oct 2026

Reinforcement Learning with Decomposed Subtasks

arXiv 2609.27035 ↗First posted 22 Sep 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 16 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

13 of 16 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification7 Jan
  2. 2025GRPO-λ: Credit Assignment improves LLM Reasoning30 Sep
  3. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  4. 2023Let's Verify Step by Step31 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 8

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  4. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  5. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  6. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 33
  7. 2026Experiential Reinforcement Learningcited by 22
  8. 2026Self-adapting Robotic Agents through Online Continual Reinforcement Learning with World Model Feedbackcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  2. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  3. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  4. 2026EPIG-Tree: Compute-Optimal Branching for Gradient-Efficient Reinforcement Learning
  5. 2026RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents
  6. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
  7. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  8. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training