mlsanityData to 7 Oct 2026

Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients

arXiv 2510.23049 ↗First posted 27 Oct 2025cs.LG cs.AIcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

This note reconciles two seemingly distinct approaches to policy gradient optimization for the Pass@K objective in reinforcement learning with verifiable rewards: (1) direct REINFORCE-style methods, and (2) advantage-shaping techniques that directly modify GRPO. We show that these are two sides of the same coin. By reverse-engineering existing advantage-shaping algorithms, we reveal that they implicitly optimize surrogate rewards. We specifically interpret practical "hard-example up-weighting" modifications to GRPO as reward-level regularization. Conversely, starting from surrogate reward objectives, we provide a simple recipe for deriving both existing and new advantage-shaping methods. This perspective provides a lens for RLVR policy gradient optimization beyond our original motivation of Pass@K.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 43 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

32 of 43 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025What is the objective of reasoning with reinforcement learning?15 Oct
  2. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning4 Oct
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization24 May
  8. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
3 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2025Group Sequence Policy Optimizationcited by 788
  7. 2020Long-tail learning via logit adjustmentcited by 605
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  9. 2019SPoC: Search-based Pseudocode to Codecited by 176
  10. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  11. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  12. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
5 more
  1. 2020Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learningcited by 46
  2. 2022Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applicationscited by 27
  3. 2021Procrustean Training for Imbalanced Deep Learningcited by 24
  4. 2019Sample Complexity of Sample Average Approximation for Conditional Stochastic Optimizationcited by 19
  5. 2023Debiasing Conditional Stochastic Optimizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 9
  2. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  3. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewardscited by 3
  4. 2026Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thoughtcited by 2
  5. 2026GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identitycited by 2
  6. 2026Epistemic Uncertainty for Test-Time Discoverycited by 0
  7. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  8. 2026Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-K Objectivecited by 0
  9. 2026Group Adaptive Clipping Policy Optimizationcited by 0
  10. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Advantage Estimates for Max@K Policy Gradients
  2. 2026Reinforcement Learning with Decomposed Subtasks
  3. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
  4. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
  5. 2026A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
  6. 2026GOPO: Policy Optimization using Ranked Rewards
  7. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  8. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training