mlsanityData to 7 Oct 2026

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

arXiv 2606.28707 ↗First posted 27 Jun 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models. However, GRPO-style advantage estimation depends on prompt-local (within-prompt-group) reward statistics and can be unstable. In particular, when all rollouts in a prompt group receive identical rewards, the within-group reward variance becomes zero, and group normalization yields zero advantages for that group, impeding learning in cold-start regimes with binary verifiers. We introduce BV-Blend, a critic-free framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. BV-Blend maintains EMA-tracked reward moments for each cluster, derives a confidence weight from a standard error of the mean (SEM) proxy, and uses this weight to blend historical and prompt-local baseline and variance statistics into a standardized advantage for PPO-style clipped updates. Experiments on verifiable reasoning benchmarks show that BV-Blend improves training stability and performance, and remains robust in regimes where group-normalized methods may stall.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 58 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

42 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  3. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning12 May
  4. 2025Learning to Reason under Off-Policy Guidance21 Apr
  5. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  6. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  10. 2025Group Relative Policy Optimization for Image Captioning3 Mar
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  12. 2023Let's Verify Step by Step31 May
2 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
16 more
  1. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  2. 2026ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrievalcited by 39
  3. 2025Table-R1: Region-based Reinforcement Learning for Table Understandingcited by 35
  4. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  5. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  6. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  7. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 24
  8. 2026ReCreate: Reasoning and Creating Domain Agents Driven by Experiencecited by 22
  9. 2026Coupling Macro Dynamics and Micro States for Long-Horizon Social Simulationcited by 14
  10. 2026NeuReasoner: Towards Explainable, Controllable, and Unified Reasoning via Mixture-of-Neuronscited by 13
  11. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 9
  12. 2026FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Modelscited by 8
  13. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  14. 2026Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generationcited by 5
  15. 2026ActorMind: Emulating Human Actor Reasoning for Speech Role-Playingcited by 4
  16. 2026GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  2. 2026A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
  3. 2026MaPP: A Unified Marginalized Posterior-Predictive Framework for Data-Efficient RLVR
  4. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
  5. 2026Rethinking Groups in Critic-Free RLVR
  6. 2026Your Group-Relative Advantage Is Biased
  7. 2025COPO: Consistency-Aware Policy Optimization
  8. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States