mlsanityData to 7 Oct 2026

Towards Understanding Specification Gaming in Reasoning Models

arXiv 2605.02269 ↗First posted 4 May 2026cs.AIcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Specification gaming is a critical failure mode of LLM agents. Despite this, there has been little systematic research into when it arises and what drives it. To address this, we build and open source a diverse suite of tasks where models can score highly by taking unintended actions. We find that all tested models exploit their specifications at non-negligible rates in most of our eight settings, including five non-coding settings. We see the highest rates of specification gaming in Grok 4 and the lowest rates in Claude models. We use our evaluation suite to study what drives specification gaming, and find that: 1. RL reasoning training substantially increases the rate at which models exploit their specifications, 2. Increasing RL reasoning budget has a weakly positive effect on exploit rate, and 3. Test-time mitigations reduce but do not eliminate the rate of specification gaming. Our results suggest that specification gaming is a fundamental challenge arising from RL reasoning training; we release our evaluation suite to support further work on this problem.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 20 of its 31 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

20 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Demonstrating specification gaming in reasoning models18 Feb
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar

Other references · 18

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2025Kimi K2: Open Agentic Intelligencecited by 523
  5. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  6. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  7. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  8. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  9. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  10. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  11. 2025Persona Features Control Emergent Misalignmentcited by 76
  12. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
6 more
  1. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 51
  2. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  3. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  4. 2025EvilGenie: A Reward Hacking Benchmarkcited by 24
  5. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  6. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 19
  2. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  3. 2026From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  2. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  3. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  4. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  5. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  6. 2026GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection
  7. 2025Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning
  8. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective