mlsanityData to 14 Oct 2026

GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control

arXiv 2508.03772 ↗First posted 5 Aug 2025cs.LG cs.AI cs.CLcited by 18

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group Relative Policy Optimization (GRPO) is a promising policy-based approach for Large Language Model alignment, yet its performance is often limited by training instability and suboptimal convergence. In this paper, we identify and analyze two main GRPO issues: (i) the token-level penalization, where valuable tokens shared across different responses receive contradictory feedback signals, leading to conflicting gradient updates that can reduce their likelihood; and (ii) the policy collapse, where negatively rewarded completions may penalize confident responses and shift model decisions toward unlikely tokens, destabilizing training process. To address these issues we introduce GTPO (Group-relative Trajectory-based Policy Optimization), which prevents conflicting gradients on valuable tokens by skipping negative updates while amplifying positive ones and filters out completions whose entropy exceeds a provable threshold, to prevent policy collapse. Unlike GRPO, GTPO does not rely on KL-divergence regularization, eliminating the need for a reference model during training, while still ensuring greater training stability and improved performance, as validated through multiple experiments on GSM8K, MATH, AIME 2024, AIME 2025 and AMC 2023.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 39 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

27 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  2. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach26 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
7 more
  1. 2025Group Sequence Policy Optimizationcited by 835
  2. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  3. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  4. 2019Trust Region-Guided Proximal Policy Optimizationcited by 36
  5. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  6. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  7. 2025Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 23
  2. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPOcited by 10
  3. 2025DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimizationcited by 9
  4. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokenscited by 9
  5. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
  6. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
  7. 2026ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learningcited by 6
  8. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradientscited by 5
  9. 2026On the Plasticity and Stability for Post-Training Large Language Modelscited by 4
  10. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 3
  11. 2025APO: Alpha-Divergence Preference Optimizationcited by 2
  12. 2026Beyond KL Divergence: Policy Optimization with Flexible Bregman Divergences for LLM Reasoningcited by 2
6 more
  1. 2026Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groupscited by 2
  2. 2026Are complicated loss functions necessary for teaching LLMs to reason?cited by 1
  3. 2026Holder Policy Optimisationcited by 0
  4. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agentscited by 0
  5. 2026Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoningcited by 0
  6. 2026ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogatecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Composing Policy Gradients and Prompt Optimization for Language Model Programs
  2. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  3. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  4. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
  5. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  6. 2025Geometric-Mean Policy Optimization
  7. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  8. 2026A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment