# LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking

arXiv: https://arxiv.org/abs/2604.15149 · HTML: https://mlsanity.com/paper/2604.15149/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 6 of 6)

- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Inductive logic programming at 30](https://mlsanity.com/paper/2102.10556/) (2021)
- [Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort](https://mlsanity.com/paper/2510.01367/) (2025)
- [SLR: Automated Synthesis for Scalable Logical Reasoning](https://mlsanity.com/paper/2506.15787/) (2025)

## Built on by (most cited first, 28 of 28)

- [SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?](https://mlsanity.com/paper/2606.07682/) (2026)
- [Before the Model Learns the Bug:Fuzzing RLVR Verifiers](https://mlsanity.com/paper/2606.01066/) (2026)
- [Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?](https://mlsanity.com/paper/2605.22148/) (2026)
- [Auditing Reward Hackability in Code RL Training Environments](https://mlsanity.com/paper/2606.16062/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR](https://mlsanity.com/paper/2607.11022/) (2026)
- [The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillation](https://mlsanity.com/paper/2607.09709/) (2026)
- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [Verifier-Backed Hard Problem Generation for Mathematical Reasoning](https://mlsanity.com/paper/2605.06660/) (2026)
- [When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVR](https://mlsanity.com/paper/2606.22043/) (2026)
- [Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets](https://mlsanity.com/paper/2608.01000/) (2026)
- [A Primer in Post-Training Reasoning Data: What We Know About How It Works](https://mlsanity.com/paper/2606.02113/) (2026)
- [Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL](https://mlsanity.com/paper/2607.16204/) (2026)
- [Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities](https://mlsanity.com/paper/2605.10810/) (2026)
- [SciR: A Controllable Benchmark for Scientific Reasoning in LLMs](https://mlsanity.com/paper/2606.13020/) (2026)
- [Predicting Task Difficulty Without Rollouts](https://mlsanity.com/paper/2608.05797/) (2026)
- [HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards](https://mlsanity.com/paper/2608.06012/) (2026)
- [An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning](https://mlsanity.com/paper/2608.17804/) (2026)
- [Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck](https://mlsanity.com/paper/2608.20362/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus](https://mlsanity.com/paper/2609.26826/) (2026)
- [When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning](https://mlsanity.com/paper/2609.33220/) (2026)
- [Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control](https://mlsanity.com/paper/2609.35677/) (2026)
- [From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents](https://mlsanity.com/paper/2609.36161/) (2026)
- [When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora](https://mlsanity.com/paper/2610.00202/) (2026)
- [NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents](https://mlsanity.com/paper/2610.03631/) (2026)
- [Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR](https://mlsanity.com/paper/2610.17824/) (2026)
- [Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following](https://mlsanity.com/paper/2610.19525/) (2026)

## Abstract (arXiv)

As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs, a new failure mode emerges: LLMs gaming verifiers. We study this phenomenon on inductive reasoning tasks, where models must induce and output logical rules. We find that RLVR-trained models systematically abandon rule induction. Instead of learning generalizable patterns (e.g., ``trains carrying red cars go east''), they enumerate instance-level labels, producing outputs that pass verifiers without capturing the relational patterns required by the task. We show that this behavior is not a failure of understanding but a form of reward hacking: imperfect verifiers that check only extensional correctness admit false positives. To detect such shortcuts, we introduce Isomorphic Perturbation Testing (IPT), which evaluates a single model output under both extensional and isomorphic verification, where the latter enforces invariance under logically isomorphic tasks. While genuine rule induction remains invariant, shortcut strategies fail. We find that shortcut behavior is specific to RLVR-trained reasoning models (e.g., GPT-5, Olmo3) and absent in non-RLVR models (e.g., GPT-4o, GPT-4.5, Ministral). Moreover, shortcut prevalence increases with task complexity and inference-time compute. In controlled training experiments, extensional verification directly induces shortcut strategies, while isomorphic verification eliminates them. These results show that RLVR can incentivize reward hacking not only through overt manipulation but also by exploiting what the verifier fails to enforce.