mlsanityData to 14 Oct 2026

Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models

arXiv 2505.23564 ↗First posted 29 May 2025cs.LG cs.AI cs.CLcited by 75

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Enhancing the reasoning capabilities of large language models effectively using reinforcement learning (RL) remains a crucial challenge. Existing approaches primarily adopt two contrasting advantage estimation granularities: token-level methods (e.g., PPO) aim to provide fine-grained advantage signals but suffer from inaccurate estimation due to difficulties in training an accurate critic model. On the other extreme, trajectory-level methods (e.g., GRPO) solely rely on a coarse-grained advantage signal from the final reward, leading to imprecise credit assignment. To address these limitations, we propose Segment Policy Optimization (SPO), a novel RL framework that leverages segment-level advantage estimation at an intermediate granularity, achieving a better balance by offering more precise credit assignment than trajectory-level methods and requiring fewer estimation points than token-level methods, enabling accurate advantage estimation based on Monte Carlo (MC) without a critic model. SPO features three components with novel strategies: (1) flexible segment partition; (2) accurate segment advantage estimation; and (3) policy optimization using segment advantages, including a novel probability-mask strategy. We further instantiate SPO for two specific scenarios: (1) SPO-chain for short chain-of-thought (CoT), featuring novel cutpoint-based partition and chain-based advantage estimation, achieving 6-12 percentage point improvements in accuracy over PPO and GRPO on GSM8K. (2) SPO-tree for long CoT, featuring novel tree-based advantage estimation, which significantly reduces the cost of MC estimation, achieving 7-11 percentage point improvements over GRPO on MATH500 under 2K and 4K context evaluation. We make our code publicly available at https://github.com/AIFrameResearch/SPO.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 40 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

32 of 40 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  6. 2025Learning to Reason at the Frontier of Learnability17 Feb
  7. 2025Process Reinforcement through Implicit Rewards3 Feb
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  11. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  12. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
6 more
  1. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  4. 2023Let's Verify Step by Step31 May
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  11. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  12. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
2 more
  1. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  2. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 56.

75 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  4. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
  5. 2025TreeRPO: Tree Relative Policy Optimizationcited by 46
  6. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 39
  7. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  8. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  9. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  10. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 12
  11. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  12. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 11
38 more
  1. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPOcited by 10
  2. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
  3. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  4. 2025ESPO: Entropy Importance Sampling Policy Optimizationcited by 9
  5. 2025TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inferencecited by 8
  6. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  7. 2025Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewardscited by 8
  8. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  9. 2026Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimizationcited by 8
  10. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  11. 2026CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimizationcited by 8
  12. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
  13. 2026Beyond Uniform Credit: Causal Credit Assignment for Policy Optimizationcited by 7
  14. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoningcited by 5
  15. 2025From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Controlcited by 4
  16. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agentscited by 4
  17. 2026Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewardscited by 4
  18. 2026Strategy-Aware Optimization Modeling with Reasoning LLMscited by 4
  19. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 4
  20. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoningcited by 4
  21. 2026Credit Assignment with Resets in Language Model Reasoningcited by 4
  22. 2026APPO: Agentic Procedural Policy Optimizationcited by 4
  23. 2026FiRe: Fine-grained Multimodal Reasoning for Enhanced Image Generationcited by 3
  24. 2026Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Usecited by 3
  25. 2026HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learningcited by 3
  26. 2026GraphPO: Graph-based Policy Optimization for Reasoning Modelscited by 3
  27. 2026CAST: Game Solvers as Turn-Level Teachers for LLM Agentscited by 3
  28. 2025Mitigating Overthinking through Reasoning Shapingcited by 2
  29. 2025The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Samplingcited by 2
  30. 2026RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimizationcited by 2
  31. 2026P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answeringcited by 2
  32. 2026GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervisioncited by 2
  33. 2026Policy Improvement Reinforcement Learningcited by 2
  34. 2026Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Modelscited by 2
  35. 2026Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flatteningcited by 2
  36. 2025OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessmentcited by 1
  37. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMscited by 1
  38. 2026Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialoguecited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training
  2. 2026Learning with a Single Rollout via Pass@k Policy Optimization
  3. 2024Adaptive Segment-level Reward: Bridging the Gap Between Action and Reward Space in Alignment
  4. 2026SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
  5. 2026PACT: From Credit Assignment to Critic Alignment
  6. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization