mlsanityData to 14 Oct 2026

A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

arXiv 2606.16154 ↗First posted 15 Jun 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse. We analyse this instability through token-level gradient dynamics, deriving a taxonomy that predicts how updates affect next-token probabilities and entropy. The taxonomy shows that stability depends jointly on the advantage sign and token distribution under the current policy. Motivated by this finding, we propose Winner Advantage Policy Optimization (WAPO), a simple online clipped policy-gradient objective that updates only on positive-advantage completions. Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability and matches or outperforms baselines across multiple model families. Full code can be found at https://github.com/layer6ai-labs/wapo.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 32 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

24 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  2. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2023Let's Verify Step by Step31 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  9. 2022CodeT: Code Generation with Generated Testscited by 384
  10. 2025Generalizing Verifiable Instruction Followingcited by 194
  11. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 180
  12. 2020Open Question Answering over Tables and Textcited by 163
5 more
  1. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  2. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 72
  3. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 41
  4. 2024MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translationcited by 19
  5. 2026Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RLcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026What are Key Factors for Updates in RL for LLM Reasoning?
  2. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
  3. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
  4. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
  5. 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
  6. 2026Rewards as Labels: Revisiting RLVR from a Classification Perspective
  7. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  8. 2025SSPO: Subsentence-level Policy Optimization