mlsanityData to 7 Oct 2026

EvilGenie: A Reward Hacking Benchmark

arXiv 2511.21654 ↗First posted 26 Nov 2025cs.LGcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

We introduce EvilGenie, a benchmark for reward hacking in programming settings. We source problems from LiveCodeBench and create an environment in which agents can easily reward hack, such as by hardcoding test cases or editing the testing files. We measure reward hacking in three ways: held out unit tests, LLM judges, and test file edit detection. We verify these methods against human review and each other. We find the LLM judge to be highly effective at detecting reward hacking in unambiguous cases, and observe only minimal improvement from the use of held out test cases. In addition to testing many models using Inspect's basicagent scaffold, we also measure reward hacking rates for three popular proprietary coding agents: OpenAI's Codex, Anthropic's Claude Code, and Google's Gemini CLI. We observe explicit reward hacking by both Codex and Claude Code, and misaligned behavior by all three agents. Our codebase can be found at https://github.com/JonathanGabor/evilgenieinspect .

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 4 of its 17 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

4 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar

Other references · 3

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 24.

24 papers.

Sort:

By area:

  1. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Usecited by 25
  2. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  3. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 24
  4. 2026ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Trainingcited by 23
  5. 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 15
  6. 2026Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scalecited by 9
  7. 2026Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loopscited by 8
  8. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AIcited by 6
  9. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  10. 2026ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agentscited by 3
  11. 2026Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflowscited by 2
  12. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 1
12 more
  1. 2026When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVRcited by 1
  2. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationscited by 1
  3. 2026Two AI Metrics Diverged: Will it Make All the Difference?cited by 0
  4. 2026Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Datacited by 0
  5. 2026Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Taskscited by 0
  6. 2026Can escalation channels redirect reward hacking toward defect disclosure?cited by 0
  7. 2026BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructurecited by 0
  8. 2026Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passescited by 0
  9. 2026CheatBench: Measuring Reward Gaming in AI Agentscited by 0
  10. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RLcited by 0
  11. 2026RH-Detect: A Unified Benchmark for Reward Hacking Detectioncited by 0
  12. 2026TRACE: Diagnosing Verifier Brittleness in Agentic Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026hacktrace: behavior-supervised detection of reward hacking during code generation
  2. 2026BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
  3. 2026Reward Hacking Challenges Oversight of Autonomous Research Agents
  4. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
  5. 2026Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories
  6. 2026MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills
  7. 2026SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
  8. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation