mlsanityData to 7 Oct 2026

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

arXiv 2601.20103 ↗First posted 27 Jan 2026cs.SE cs.AI cs.LGcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 78 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

40 of 78 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025EvilGenie: A Reward Hacking Benchmark26 Nov
  2. 2025Natural Emergent Misalignment from Reward Hacking in Production RL23 Nov
  3. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs24 Aug
  4. 2025Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinement24 Jul
  5. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests8 Jul
  6. 2024Feedback Loops With Language Models Drive In-Context Reward Hacking9 Feb
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  10. 2025Kimi K2: Open Agentic Intelligencecited by 523
  11. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  12. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
21 more
  1. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  2. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  3. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  4. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  5. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  6. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  7. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  8. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  9. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 51
  10. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 51
  11. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  12. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  13. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  14. 2022ThinkSum: Probabilistic reasoning over sets using large language modelscited by 23
  15. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  16. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  17. 2024Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learningcited by 7
  18. 2024Large Language Models for Anomaly Detection in Computational Workflows: from Supervised Fine-Tuning to In-Context Learningcited by 6
  19. 2024Optimization of Retrieval-Augmented Generation Context with Outlier Detectioncited by 1
  20. 2025ICAD-LLM: One-for-All Anomaly Detection via In-Context Learning with Large Language Modelscited by 1
  21. 2025SetAD: Semi-Supervised Anomaly Learning in Contextual Setscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  2. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  3. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 19
  4. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  5. 2026Detecting Safety Violations Across Many Agent Tracescited by 11
  6. 2026Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scalecited by 9
  7. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  8. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
  9. 2026Auditing Harness Tampering in Self-Improving Agentscited by 3
  10. 2026Likelihood Hacking in Probabilistic Program Synthesiscited by 2
  11. 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilitiescited by 2
  12. 2026Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedbackcited by 1
3 more
  1. 2026When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVRcited by 1
  2. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RLcited by 0
  3. 2026RH-Detect: A Unified Benchmark for Reward Hacking Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation
  2. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
  3. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
  4. 2026hacktrace: behavior-supervised detection of reward hacking during code generation
  5. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  6. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  7. 2026Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
  8. 2026Multimodal Reward Hacking in Reinforcement Learning