mlsanityData to 14 Oct 2026

Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers

arXiv 2510.00915 ↗First posted 1 Oct 2025cs.LG cs.AIcited by 45

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) replaces costly human labeling with automated verifiers. To reduce verifier hacking, many RLVR systems binarize rewards to 0,1, but imperfect verifiers inevitably introduce false negatives (rejecting correct answers) and false positives (accepting incorrect ones). We formalize verifier unreliability as a stochastic reward channel with asymmetric noise rates ρ0 and ρ1 -- the FP rate and the FN rate, respectively. From this abstraction we derive two lightweight corrections: (i) a backward correction that yields an unbiased surrogate reward and thus an unbiased policy-gradient estimator in expectation, and (ii) a forward correction that reweights score-function terms so the expected update aligns with the clean gradient direction and requires only the FN rate. We implement both as lightweight hooks in a group relative policy optimization pipeline, both corrections improve RLVR for math reasoning under synthetic and real verifier noise, with the forward variant being more stable under heavier noise. Finally, an appeals mechanism with a lightweight LLM verifier estimates the FN rate online and further improves performance.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 46 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

35 of 46 references.

References · 35

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2024A Survey on LLM-as-a-Judgecited by 1,423
23 more
  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  5. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  6. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  7. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  8. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  9. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  10. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  11. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  12. 2017Noisy Networks for Explorationcited by 266
  13. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  14. 2017Parameter Space Noise for Explorationcited by 161
  15. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  16. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 110
  17. 2021Provably End-to-end Label-Noise Learning without Anchor Pointscited by 97
  18. 2018Reinforcement Learning with Perturbed Rewardscited by 79
  19. 2025One Token to Fool LLM-as-a-Judgecited by 60
  20. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  21. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domainscited by 24
  22. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoningcited by 21
  23. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 43.

45 papers.

Sort:

By area:

  1. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 13
  2. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 11
  3. 2026Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clusteringcited by 7
  4. 2026Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVRcited by 7
  5. 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewardscited by 5
  6. 2026Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?cited by 5
  7. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  8. 2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loopcited by 4
  9. 2026Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacescited by 4
  10. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 4
  11. 2026High-Dimensional Statistics: Reflections on Progress and Open Problemscited by 4
  12. 2026SocraticPO: Policy Optimization via Interactive Guidancecited by 4
33 more
  1. 2026When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVRcited by 4
  2. 2026Probing RLVR training instability through the lens of objective-level hackingcited by 3
  3. 2026SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregationcited by 3
  4. 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearchcited by 3
  5. 2026The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillationcited by 3
  6. 2026Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVRcited by 3
  7. 2026Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairingcited by 2
  8. 2026Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Modelscited by 2
  9. 2026Quantifying Empirical Compute-Supervision Tradeoffs in RLVRcited by 2
  10. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  11. 2026MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agentscited by 2
  12. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Setscited by 2
  13. 2026VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Followingcited by 2
  14. 2026Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rolloutscited by 2
  15. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewardscited by 1
  16. 2026Reinforcement Learning for Computer-Use Agents with Autonomous Evaluationcited by 1
  17. 2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
  18. 2025Do Math Reasoning LLMs Help Predict the Impact of Public Transit Events?cited by 0
  19. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  20. 2026Safe Bilevel Delegation (SBD): A Formal Framework for Runtime Delegation Safety in Multi-Agent Systemscited by 0
  21. 2026On Training in Imaginationcited by 0
  22. 2026Constraint-First Reasoning: A Training-Free Protocol for Exploiting Answer-Space Constraints in Mathematical Problem Solvingcited by 0
  23. 2026HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewardscited by 0
  24. 2026Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneckcited by 0
  25. 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
  26. 2026Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verificationcited by 0
  27. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  28. 2026WEFT: Scaling Tool-Use Post-Training for General-Purpose Agentscited by 0
  29. 2026S3: Spectral Null-Space Swap Makes Reasoning Models Efficientcited by 0
  30. 2026When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corporacited by 0
  31. 2026SquidAgent: Parallelize Wisely, Coordinate Efficientlycited by 0
  32. 2026Finding Blind Spots in AppWorld and WorkArena Task Verifierscited by 0
  33. 2026Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
  2. 2026Before the Model Learns the Bug:Fuzzing RLVR Verifiers
  3. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
  4. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  5. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
  6. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
  7. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards
  8. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity