mlsanityData to 14 Oct 2026

STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking

arXiv 2609.36900 ↗First posted 29 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reward hacking occurs when policy optimization exploits a brittle reward interface or an overly permissive proxy objective, improving the training score without improving the underlying response quality. This phenomenon is amplified in group-relative policy optimization: an unsupported reward can shift the group baseline and alter the updates of other rollouts, while post-hoc or purely relative weighting cannot represent group-wide uncertainty. We propose Self-Tuned Anchored Reliability Group-Relative Policy Optimization (STAR-GRPO), a reliability-first advantage estimator based on paired assessments of the same rollout. STAR separates the quality signal from its learning influence: score disagreement determines rollout reliability, relative reliability enters a self-tuned robust location--scale fit before group normalization, and absolute group reliability attenuates the resulting bounded advantage. The analysis establishes coordinate and second-moment bounds, characterizes exact centering through the weighted location equation, and gives reliability-dependent attenuation guarantees for outlying rewards. We evaluate STAR-GRPO in two complementary reward-hacking regimes. In token-interface exploitation, STAR prevents runaway optimization of the deployed-interface score while improving the canonical quality signal. In rubric-proxy overoptimization for medical reasoning, STAR improves independent semantic evaluation, narrows the proxy--judge discrepancy, and reduces overclaim while optimizing the same task proxy. Together, these results show that reliability-first normalization offers a principled way to limit unsupported reward influence on both group baselines and policy updates, while retaining the task reward as the optimization target.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 34 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

27 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  11. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  12. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
11 more
  1. 2022Conformal prediction beyond exchangeabilitycited by 355
  2. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  3. 2019Mean estimation and regression under heavy-tailed distributions--a surveycited by 136
  4. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  5. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 66
  6. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  7. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  8. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 7
  9. 2021Do we need to estimate the variance in robust mean estimation?cited by 6
  10. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
  11. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems
  2. 2026GOPO: Policy Optimization using Ranked Rewards
  3. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
  4. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
  5. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
  6. 2026Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation
  7. 2025GRPO is Secretly a Process Reward Model
  8. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works