mlsanityData to 7 Oct 2026

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

arXiv 2603.10535 ↗First posted 11 Mar 2026cs.LG cs.CLcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR3), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR3~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 47 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

39 of 47 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
27 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  4. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  5. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  7. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 278
  8. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  9. 2025Training Language Models to Reason Efficientlycited by 242
  10. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  11. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  12. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  13. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  14. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  15. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  16. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  17. 2025Efficient Reasoning Models: A Surveycited by 93
  18. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  19. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  20. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  21. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 74
  22. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  23. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  24. 2025Optimizing Length Compression in Large Reasoning Modelscited by 37
  25. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 34
  26. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  27. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compressioncited by 2
  2. 2026Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasonerscited by 2
  3. 2026Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Constructioncited by 0
  4. 2026GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMscited by 0
  5. 2026CROCODIL: Cross-Model Code Editing with LLMscited by 0
  6. 2026The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewardscited by 0
  7. 2026Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
  2. 2026Constrained Group Relative Policy Optimization
  3. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  4. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  5. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  6. 2026On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
  7. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  8. 2026Group Adaptive Clipping Policy Optimization