mlsanityData to 7 Oct 2026

Cheap Reward Hacking Detection

arXiv 2606.08893 ↗First posted 8 Jun 2026cs.LG cs.AI cs.CRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

A small transformer encoder is trained to map Terminal-Wrench trajectories onto a unit sphere where embedding distance approximates the L1 distance between reward and metadata signals. A linear probe on top of that embedding detects reward hacking on the cleaned test split with AUC 0.9467 and TPR@5%FPR 0.8296, matching the TW sanitized LLM-as-judge AUC (0.9510 on the cleaned split) and exceeding its TPR@5%FPR (0.7130 vs 0.8296) on the same information condition, at roughly four orders of magnitude lower per-trajectory cost. The encoder is not a pure behavior reader: stripping natural-language reasoning from its input at probe time drops AUC to 0.6213.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 20 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

12 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 11

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  6. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  7. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  8. 2018Categorizing Variants of Goodhart's Lawcited by 132
  9. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  10. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  11. 2026Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectoriescited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RH-Detect: A Unified Benchmark for Reward Hacking Detection
  2. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
  3. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
  4. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  5. 2026hacktrace: behavior-supervised detection of reward hacking during code generation
  6. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprints
  7. 2026Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
  8. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs