mlsanityData to 7 Oct 2026

Inducing Emergent Misalignment from Reward Hacks with Iterative DPO

arXiv 2609.06649 ↗First posted 6 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alternative, we propose studying emergent misalignment from iterative DPO, which preserves important properties of RLVR while reducing costs and enabling training on popular finetuning APIs. In practice, we find that training GPT-4.1 with iterative DPO on a single-turn reward hacking environment induces covert misaligned power-seeking and alignment faking, the first openly available (semi)-online training pipeline to induce these concerning forms of misalignment. We also find that training Qwen2.5-32B-Instruct with the same pipeline induces both misalignment and improved instruction following accuracy, showing that iterative DPO can be used as a testbed for selective generalization. Overall, we think iterative DPO can help democratize and accelerate the study of emergent misalignment from RLVR.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 45 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

20 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment6 Oct
  2. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs24 Aug

Other references · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  10. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  11. 2024Frontier Models are Capable of In-context Schemingcited by 252
  12. 2025Demonstrating specification gaming in reasoning modelscited by 48
6 more
  1. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  2. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  3. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 25
  4. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 19
  5. 2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 9
  6. 2026Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoorscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  2. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
  3. 2026Multimodal Reward Hacking in Reinforcement Learning
  4. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
  5. 2025Natural Emergent Misalignment from Reward Hacking in Production RL
  6. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
  7. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  8. 2026IR3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking