# Quantifying Empirical Compute-Supervision Tradeoffs in RLVR

arXiv: https://arxiv.org/abs/2605.25252 · HTML: https://mlsanity.com/paper/2605.25252/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 17 of 17)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [RLTF: Reinforcement Learning from Unit Test Feedback](https://mlsanity.com/paper/2307.04349/) (2023)
- [The Limits of Inference Scaling Through Resampling](https://mlsanity.com/paper/2411.17501/) (2024)
- [Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers](https://mlsanity.com/paper/2510.00915/) (2025)
- [Reward-Robust RLHF in LLMs](https://mlsanity.com/paper/2409.15360/) (2024)
- [An Imperfect Verifier is Good Enough: Learning with Noisy Rewards](https://mlsanity.com/paper/2604.07666/) (2026)

## Built on by (most cited first, 2 of 2)

- [Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery](https://mlsanity.com/paper/2609.19164/) (2026)
- [Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models](https://mlsanity.com/paper/2609.27166/) (2026)

## Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training language models, but in practice, verifiers are rarely perfect. Recent theoretical work predicts that verifier noise affects the rate of learning but not its final outcome, implying that sufficient compute should close any gap induced by imperfect supervision. We test this prediction empirically by post-training Qwen2.5 (0.5B, 1.5B) with GRPO on GSM8K while injecting controlled false-positive and false-negative noise into the binary correctness signal, and varying rollouts per prompt as a compute axis. In practice, the gap in validation accuracy persists under substantial compute scaling, with returns to compute that are sharply diminishing. We further find a structural asymmetry where false negatives monotonically degrade performance more quickly than false positives. These findings suggest verifier quality and training compute are not interchangeable, and that reducing false negatives is a more effective lever than scaling compute alone.