mlsanityData to 14 Oct 2026

Measuring Reward-Seeking via Contrastive Belief Updates

arXiv 2607.18966 ↗First posted 21 Jul 2026cs.AI cs.CL cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective. This "reward-seeking" is difficult to measure because a model that pursues the grader's judgment and one that pursues the intended objective behave identically whenever the grader rewards the intended behavior. We measure reward-seeking using Contrastive Synthetic Document Finetuning to change a model's beliefs about what the grader rewards, putting those beliefs in conflict with what users or developers want, and measuring the rate at which the model adopts each party's preferred behavior. Applied to intermediate checkpoints of a capabilities-focused OpenAI o3 RL run, without safety training, we find that these checkpoints often side with grader preferences over those of users or developers on coding and alignment tasks. This tendency to side with the grader trends upward throughout RL training. For example, in an environment that forces a choice between keeping a promise to a supervisor and breaking it to complete the task, a late capabilities-focused o3 checkpoint breaks the promise 87% of the time when SDF documents say the grader rewards task completion, versus 9% when they say it rewards honesty (a choice its chain-of-thought often makes explicit). An earlier checkpoint is far less sensitive (40% vs. 24%). Our method also generalizes to reward-hacking models. A model organism trained to reward-hack (gpt-oss-120b) is more than twice as sensitive to grader preferences as the unmodified model, with the mean behavioral shift in favor of the grader rising from 33% to 86%. These results indicate that RL can increase reward-seeking over the course of training, producing models that may act against their developers' intentions when they believe that doing so leads to higher reward.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 53 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

27 of 53 references.

References · 27

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  4. 2024Alignment faking in large language modelscited by 360
  5. 2024Frontier Models are Capable of In-context Schemingcited by 287
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  8. 2022Defining and Characterizing Reward Hackingcited by 203
  9. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  10. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  11. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  12. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
15 more
  1. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  2. 2025Persona Features Control Emergent Misalignmentcited by 86
  3. 2025Stress Testing Deliberative Alignment for Anti-Scheming Trainingcited by 86
  4. 2024Prover-Verifier Games improve legibility of LLM outputscited by 85
  5. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  6. 2025Auditing language models for hidden objectivescited by 83
  7. 2025Demonstrating specification gaming in reasoning modelscited by 56
  8. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  9. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 43
  10. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  11. 2025Steering Evaluation-Aware Language Models to Act Like They Are Deployedcited by 35
  12. 2025Believe It or Not: How Deeply do LLMs Believe Implanted Facts?cited by 24
  13. 2025Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMscited by 22
  14. 2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 17
  15. 2025Evaluation Awareness Scales Predictably in Open-Weights Large Language Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Quantifying Overclaiming Propensity in Frontier LLM Agentscited by 4
  2. 2026Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Taskscited by 3
  3. 2026Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Bestcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RewardBench 2: Advancing Reward Model Evaluation
  2. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
  3. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction following
  4. 2026Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability
  5. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  8. 2019Fine-Tuning Language Models from Human Preferences