mlsanityData to 14 Oct 2026

On Advantage Estimates for Max@K Policy Gradients

arXiv 2606.06080 ↗First posted 4 Jun 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning with verifiable rewards is widely used for post-training reasoning models, but sparse outcome rewards make exploration difficult. A complementary approach is to optimize inference-time objectives such as pass@K and max@K directly, yet existing policy-gradient estimators for these objectives use different signals, baselines, and normalizations, making their relationships unclear. We study this issue through baseline design and advantage centering. Starting from the advantage estimator of a leading method in the field, we show that it is policy-gradient unbiased but yields a non-centered advantage. We then introduce a Leave-Two-Out baseline that preserves policy-gradient unbiasedness while making realized batch advantages exactly centered. The resulting method, MaxPO, has an efficient quadratic-time implementation and integrates naturally into group-based RL for LLM post-training. We further derive the canonical finite-batch advantage for max@K, providing a unified view of existing advantage estimators. Empirically, we verify that the L2O baseline reduces gradient variance and outperforms non-centered alternatives.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 71 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

60 of 71 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning17 Dec
  2. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models8 Dec
  3. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization2 Oct
  4. 2025Learning to Reason as Action Abstractions with Scalable Mid-Training RL30 Sep
  5. 2025Rethinking Entropy Regularization in Large Reasoning Models29 Sep
  6. 2025Emergent Slow Thinking in LLMs as Inverse Tree Freezing28 Sep
  7. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs25 Sep
  8. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  9. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  10. 2025First Return, Entropy-Eliciting Explore9 Jul
  11. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  12. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
10 more
  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  2. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  5. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  7. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
26 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  3. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  4. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  5. 2016Variational inference for Monte Carlo objectivescited by 84
  6. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  7. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 79
  8. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 68
  9. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  10. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  11. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  12. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  13. 2025Rethinking Reflection in Pre-Trainingcited by 43
  14. 2025On Entropy Control in LLM-RL Algorithmscited by 40
  15. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
  16. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 37
  17. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  18. 2025The Coverage Principle: How Pre-Training Enables Post-Trainingcited by 30
  19. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 26
  20. 2023The Role of Baselines in Policy Gradient Optimizationcited by 25
  21. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  22. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  23. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  24. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  25. 2019Total stochastic gradient algorithms and applications in reinforcement learningcited by 7
  26. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  2. 2026Theoretical Foundations of max@k Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
  2. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
  3. 2026Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training
  4. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
  5. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  6. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
  7. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
  8. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards