mlsanityData to 7 Oct 2026

A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control

arXiv 2610.03458 ↗First posted 2 Oct 2026cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Post-training with verifiable rewards can induce reward hacking, motivating the use of monitors within the training objective rather than solely for offline auditing. We show that a low monitor readout does not identify whether such an intervention controls behavior. In a code-generation environment whose dominant exploit is available at the start of the reasoning trace, we train policies against three monitors that pass the same offline gate: an in-domain activation probe and two penalties conditioned on how early the policy commits to its own final answer. The probe score is at its numerical floor from the first recorded training step, and the trained-score median is zero for every prefix-trained run at the endpoint. These readouts estimate different quantities, and we do not compare their scales; within each monitor family, however, low values do not establish behavioral control. Within one fixed configuration, prefix-trained runs with the same zero-median trained score range, by seed alone, from a mixed regime with a low hacking share to near-pure reward hacking. All probe runs reach the hacking regime, but their floor-level readout reflects a mismatch between the position where the probe was validated and the position where it was read during training, not a second instance of this ambiguity. Text-level analysis identifies a prefix failure mode: generic planning and filler shells postpone the exploit past the cut without eliminating it from the final output. Low measured commitment therefore does not distinguish a low hacking share from delayed commitment to the exploit. Offline discrimination and low monitor-aligned readouts are insufficient evidence of behavioral control; an out-of-band behavioral check is required. We characterize the endpoint readout, not its evolution. Code is available at https://github.com/zhezhou1106/spoof-cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. First posted on arXiv 2 Oct 2026.

Builds on

No reference list yet.

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation
  2. 2026Training on Documents About Monitoring Leads to CoT Obfuscation
  3. 2026A False Average: Pooled CoT-Monitor Accuracy Conceals a Reasoning-Dependent Fragility
  4. 2026hacktrace: behavior-supervised detection of reward hacking during code generation
  5. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
  6. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort
  7. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
  8. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering