mlsanityData to 7 Oct 2026

GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy

arXiv 2508.04349 ↗First posted 6 Aug 2025cs.CL cs.AIcited by 18

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) is pivotal for enhancing Large Language Model (LLM) reasoning, yet mainstream algorithms such as GRPO and DAPO remain constrained by a coarse-grained credit assignment paradigm, where all tokens within the same response receive the identical reward. In this paper, we propose Dynamic Entropy Weighting, systematically define entropy-based weight ratios fracHi,tsumk=1n Hk,t and similar variants to redistribute rewards and get fine-grained rewards through two new algorithms: Group Token Policy Optimization (GTPO), which assigns an entropy-weighted reward to each token and synthesizes token-specific advantage function to drive the model toward optimal path, and the analogous algorithm Sequence-Level GRPO (GRPO-S), which extends this design to the sequence level and exhibits superior stability in long Chain-of-Thought (CoT) reasoning tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 38 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

33 of 38 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  5. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  8. 2025Process Reinforcement through Implicit Rewards3 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2023Let's Verify Step by Step31 May
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2022Solving math word problems with process- and outcome-based feedback25 Nov
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
8 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  5. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  7. 2024Direct Preference Optimization with an Offsetcited by 122
  8. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 15.

18 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  3. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoningcited by 12
  4. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancementcited by 2
  5. 2026HERO: Hierarchical Evidential Reasoning Optimization for Radiology Report Generation via Reason-then-Summarizecited by 2
  6. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priorscited by 2
  7. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  8. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoningcited by 1
  9. 2026SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoningcited by 1
  10. 2026Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memescited by 1
  11. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  12. 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoningcited by 1
6 more
  1. 2026Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinkingcited by 1
  2. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1
  3. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  4. 2026Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purificationcited by 0
  5. 2026Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimizationcited by 0
  6. 2026BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
  2. 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
  3. 2026Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
  4. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
  5. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
  6. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
  7. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
  8. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO