mlsanityData to 13 Oct 2026

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

arXiv 2609.35677 ↗First posted 28 Sep 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves selective control: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 38 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

32 of 38 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Debate Training Reduces Reward Hacking in RLAIF18 Aug
  2. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning3 Jun
  3. 2026Reward Hacking in Rubric-Based Reinforcement Learning12 May
  4. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprints17 Apr
  5. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR7 Mar
  6. 2026IR3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking23 Feb
  7. 2025Natural Emergent Misalignment from Reward Hacking in Production RL23 Nov
  8. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar
  9. 2024Spontaneous Reward Hacking in Iterative Self-Refinement5 Jul
  10. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking5 Mar
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 21

  1. 2023Let's Verify Step by Stepcited by 3,672
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,077
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,020
  4. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 226
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 196
  6. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 168
  7. 2021Consequences of Misaligned AIcited by 79
  8. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  9. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 58
  10. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 43
  11. 2023Goodhart's Law in Reinforcement Learningcited by 38
  12. 2025Inference-Time Reward Hacking in Large Language Modelscited by 33
9 more
  1. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 26
  2. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 26
  3. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 21
  4. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  5. 2026Patterning: The Dual of Interpretabilitycited by 6
  6. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 4
  7. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 4
  8. 2017Developing a Correlation Indices to Identify Coordinated Cyber-Attacks on Power Gridscited by 2
  9. 2026Interpreting Reinforcement Learning Agents with Susceptibilitiescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
  2. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards
  3. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  4. 2026Quantifying Empirical Compute-Supervision Tradeoffs in RLVR
  5. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
  6. 2026On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
  7. 2025Auditing Data Membership in Reinforcement Learning With Verifiable Rewards
  8. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance