# Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers

arXiv: https://arxiv.org/abs/2510.00915 · HTML: https://mlsanity.com/paper/2510.00915/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 35)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Optimization-based Prompt Injection Attack to LLM-as-a-Judge](https://mlsanity.com/paper/2403.17710/) (2024)
- [Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification](https://mlsanity.com/paper/2503.06639/) (2025)
- [Provably End-to-end Label-Noise Learning without Anchor Points](https://mlsanity.com/paper/2102.02400/) (2021)
- [Reinforcement Learning with Perturbed Rewards](https://mlsanity.com/paper/1810.01032/) (2018)

## Built on by (most cited first, 30 of 45)

- [An Imperfect Verifier is Good Enough: Learning with Noisy Rewards](https://mlsanity.com/paper/2604.07666/) (2026)
- [Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR](https://mlsanity.com/paper/2605.20164/) (2026)
- [Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering](https://mlsanity.com/paper/2601.08427/) (2026)
- [Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR](https://mlsanity.com/paper/2605.02909/) (2026)
- [Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2603.16140/) (2026)
- [Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?](https://mlsanity.com/paper/2605.22148/) (2026)
- [Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories](https://mlsanity.com/paper/2511.07368/) (2025)
- [MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop](https://mlsanity.com/paper/2601.22900/) (2026)
- [Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspaces](https://mlsanity.com/paper/2603.06713/) (2026)
- [DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay](https://mlsanity.com/paper/2603.16157/) (2026)
- [High-Dimensional Statistics: Reflections on Progress and Open Problems](https://mlsanity.com/paper/2605.05076/) (2026)
- [SocraticPO: Policy Optimization via Interactive Guidance](https://mlsanity.com/paper/2606.09887/) (2026)
- [When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR](https://mlsanity.com/paper/2607.11022/) (2026)
- [Probing RLVR training instability through the lens of objective-level hacking](https://mlsanity.com/paper/2602.01103/) (2026)
- [SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation](https://mlsanity.com/paper/2603.16161/) (2026)
- [VistaHop: Benchmarking Long-Horizon Visual DeepSearch](https://mlsanity.com/paper/2606.03273/) (2026)
- [The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillation](https://mlsanity.com/paper/2607.09709/) (2026)
- [Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR](https://mlsanity.com/paper/2609.01354/) (2026)
- [Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing](https://mlsanity.com/paper/2602.03452/) (2026)
- [Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models](https://mlsanity.com/paper/2603.27375/) (2026)
- [Quantifying Empirical Compute-Supervision Tradeoffs in RLVR](https://mlsanity.com/paper/2605.25252/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents](https://mlsanity.com/paper/2607.24097/) (2026)
- [Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets](https://mlsanity.com/paper/2608.01000/) (2026)
- [VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following](https://mlsanity.com/paper/2608.26013/) (2026)
- [Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts](https://mlsanity.com/paper/2609.06386/) (2026)
- [Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards](https://mlsanity.com/paper/2601.04411/) (2026)
- [Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation](https://mlsanity.com/paper/2606.24515/) (2026)
- [TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI](https://mlsanity.com/paper/2607.22465/) (2026)
- [Do Math Reasoning LLMs Help Predict the Impact of Public Transit Events?](https://mlsanity.com/paper/2511.00808/) (2025)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) replaces costly human labeling with automated verifiers. To reduce verifier hacking, many RLVR systems binarize rewards to 0,1, but imperfect verifiers inevitably introduce false negatives (rejecting correct answers) and false positives (accepting incorrect ones). We formalize verifier unreliability as a stochastic reward channel with asymmetric noise rates ρ0 and ρ1 -- the FP rate and the FN rate, respectively. From this abstraction we derive two lightweight corrections: (i) a backward correction that yields an unbiased surrogate reward and thus an unbiased policy-gradient estimator in expectation, and (ii) a forward correction that reweights score-function terms so the expected update aligns with the clean gradient direction and requires only the FN rate. We implement both as lightweight hooks in a group relative policy optimization pipeline, both corrections improve RLVR for math reasoning under synthetic and real verifier noise, with the forward variant being more stable under heavier noise. Finally, an appeals mechanism with a lightweight LLM verifier estimates the FN rate online and further improves performance.