# Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards

arXiv: https://arxiv.org/abs/2511.03710 · HTML: https://mlsanity.com/paper/2511.03710/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- First posted on arXiv 5 Nov 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Built on by (most cited first, 16 of 16)

- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization](https://mlsanity.com/paper/2602.05165/) (2026)
- [LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation](https://mlsanity.com/paper/2602.12351/) (2026)
- [ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement](https://mlsanity.com/paper/2604.01591/) (2026)
- [Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States](https://mlsanity.com/paper/2605.07579/) (2026)
- [Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning](https://mlsanity.com/paper/2604.28005/) (2026)
- [PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning](https://mlsanity.com/paper/2602.03352/) (2026)
- [BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning](https://mlsanity.com/paper/2605.27293/) (2026)
- [Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery](https://mlsanity.com/paper/2609.19164/) (2026)
- [Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspective](https://mlsanity.com/paper/2601.22532/) (2026)
- [Annealed Softmax Greedy in Many-Armed Bayesian Bandits](https://mlsanity.com/paper/2605.31034/) (2026)
- [Tandem Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2606.28166/) (2026)
- [Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification](https://mlsanity.com/paper/2609.07148/) (2026)
- [ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control](https://mlsanity.com/paper/2609.34374/) (2026)
- [AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.03223/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for post-training large reasoning models (LRMs) using policy-gradient methods such as GRPO. To stabilize training, these methods typically center trajectory rewards by subtracting the empirical mean reward for each prompt. Statistically, this centering acts as a control variate (baseline), reducing the variance of the policy-gradient estimator. In practice, the mean reward is estimated using per-prompt empirical averages computed from the generations for each prompt in a batch. Motivated by Stein's paradox, we propose shrinkage estimators that combine per-prompt and across-prompt means to improve per-prompt mean estimation accuracy, especially in the low-generation regime typical of RLVR. Theoretically, we construct a shrinkage-based baseline that provably yields lower-variance policy-gradient estimators across algorithms. Our baseline is a drop-in replacement for standard per-prompt mean baselines and requires no additional hyperparameters or computation. Empirically, shrinkage baselines consistently outperform empirical-mean baselines, producing lower-variance gradient updates and improved training stability.