mlsanityData to 7 Oct 2026

Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning

arXiv 2509.22611 ↗First posted 26 Sep 2025cs.LG cs.AIcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between {entropy collapse} and {entropy explosion}. We trace both hazards to the mean baseline used in value-free RL (e.g., GRPO and DAPO), which improperly penalizes negative-advantage samples under reward outliers. We propose {Quantile Advantage Estimation} (QAE), replacing the mean with a group-wise K-quantile baseline. QAE induces a response-level, two-regime gate: on hard queries (p <= 1 - K) it reinforces rare successes, while on easy queries (p > 1 - K) it targets remaining failures. Under first-order softmax updates, we prove {two-sided entropy safety}, giving lower and upper bounds on one-step entropy change that curb explosion and prevent collapse. Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned K, roughly 80% of responses receive zero advantage), and yields sustained pass@1 gains on Qwen3-8B/14B-Base across AIME 2024/2025 and AMC 2023. These results identify {baseline design} -- rather than token-level heuristics -- as the primary mechanism for scaling RLVR.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 38 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

33 of 38 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  2. 2025rStar2-Agent: Agentic Reasoning Technical Report28 Aug
  3. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning23 Aug
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  7. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  10. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization18 May
  11. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
7 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2025Group Sequence Policy Optimizationcited by 788
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2018Understanding the impact of entropy on policy optimizationcited by 171
  8. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  9. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  10. 2025Understanding Aha Moments: from External Observations to Internal Mechanismscited by 48
  11. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  12. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
2 more
  1. 2021Target Entropy Annealing for Discrete Soft Actor-Criticcited by 10
  2. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rolloutscited by 5
  2. 2026When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewardscited by 4
  3. 2026ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RLcited by 4
  4. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 2
  5. 2026Experience Augmented Policy Optimization for LLM Reasoningcited by 2
  6. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Modelscited by 1
  7. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
  8. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  9. 2026EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QAcited by 0
  10. 2026Spurious Advantage Hidden in GRPOcited by 0
  11. 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
  2. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  3. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
  4. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
  5. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  6. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  7. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  8. 2026On Advantage Estimates for Max@K Policy Gradients