mlsanityData to 14 Oct 2026

TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback

arXiv 2508.02833 ↗First posted 4 Aug 2025cs.LGcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group Relative Policy Optimization (GRPO) trains language models without a value critic using rewards centered within sampled response groups. We study how importance weighting, clipping, and length normalization affect its stochastic updates and propose Trajectory-level Importance-Corrected GRPO (TIC-GRPO), combining an upper importance-ratio cap with a full-trajectory likelihood ratio. Under common maximum-length normalization, a trajectory change of measure gives a score second moment proportional to the response horizon. We propagate this estimate through the actual empirical baseline and all inner updates. With separately optimized constant steps, the sampling term in TIC's stationarity bound has a linear horizon coefficient, versus order three halves for token-level GRPO2. Matching upper and lower bounds separate the capped methods' worst-case response-level variances by a factor T. The bounds track prompt count, response-group size, vocabulary size, reward scale, and score regularity; per-response-normalized GRPO also retains a length-covariance term. Experiments at two Qwen3 scales on four reasoning and coding benchmarks evaluate both mechanisms in sampled-token-normalized training.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 45 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

26 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2023Let's Verify Step by Step31 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2024Alignment faking in large language modelscited by 360
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  10. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  11. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
  12. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
8 more
  1. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  2. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  3. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  4. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 17
  5. 2020Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Samplingcited by 15
  6. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  7. 2024A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGDcited by 9
  8. 2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoningcited by 12
  2. 2025Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RLcited by 7
  3. 2026When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Modelscited by 6
  4. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 3
  5. 2026SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregationcited by 3
  6. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  7. 2026FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Datacited by 1
  8. 2025In2x at WMT25 Translation Taskcited by 0
  9. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0
  10. 2026Spurious Advantage Hidden in GRPOcited by 0
  11. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  2. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  3. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  4. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
  5. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  6. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  7. 2026Gradient Extrapolation-Based Policy Optimization
  8. 2026GOPO: Policy Optimization using Ranked Rewards