Data to 14 Oct 2026

School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs

arXiv 2508.17511 ↗First posted 24 Aug 2025cs.AIcited by 66

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reward hacking--where agents exploit flaws in imperfect reward functions rather than performing tasks as intended--poses risks for AI alignment. Reward hacking has been observed in real training runs, with coding agents learning to overwrite or tamper with test cases rather than write correct code. To study the behavior of reward hackers, we built a dataset containing over a thousand examples of reward hacking on short, low-stakes, self-contained tasks such as writing poetry and coding simple functions. We used supervised fine-tuning to train models (GPT-4.1, GPT-4.1-mini, Qwen3-32B, Qwen3-8B) to reward hack on these tasks. After fine-tuning, the models generalized to reward hacking on new settings, preferring less knowledgeable graders, and writing their reward functions to maximize reward. Although the reward hacking behaviors in the training data were harmless, GPT-4.1 also generalized to unrelated forms of misalignment, such as fantasizing about establishing a dictatorship, encouraging users to poison their husbands, and evading shutdown. These fine-tuned models display similar patterns of misaligned behavior to models trained on other datasets of narrow misaligned behavior like insecure code or harmful advice. Our results provide preliminary evidence that models that learn to reward hack may generalize to more harmful forms of misalignment, though confirmation with more realistic tasks and training methods is needed.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 23 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

12 of 23 references.

Closest ancestors · 1

Same direction, newest first

2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar

Other references · 11

2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2021Program Synthesis with Large Language Modelscited by 3,637
2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
2025Model Organisms for Emergent Misalignmentcited by 91
2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thoughtcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 59.

66 papers.

Sort:

By area:

2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Usecited by 31
2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 18
2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 17
2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 17
2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 14
2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
2026Adaptive Milestone Reward for GUI Agentscited by 12
2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 12
2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 11

38 more

2026Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scalecited by 10
2026Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loopscited by 9
2025LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactionscited by 8
2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 7
2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 7
2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 5
2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 4
2026Hodoscope: Unsupervised Monitoring for AI Misbehaviorscited by 3
2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 3
2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agentscited by 3
2026Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoorscited by 3
2026Emergent Misalignment Recruits a Pre-existing Persona Subspacecited by 3
2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizerscited by 2
2026SteerRM: Debiasing Reward Models via Sparse Autoencoderscited by 2
2026Likelihood Hacking in Probabilistic Program Synthesiscited by 2
2026Consistency Training Can Entrench Misalignmentcited by 2
2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
2026An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?cited by 2
2026Filtering Harmful Actions Isn't Enough: Phantom Transfer in Agentic SDFcited by 2
2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationscited by 2
2025Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignmentcited by 1
2026Generalization to Political Beliefs from Fine-Tuning on Sports Team Preferencescited by 1
2026Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineeringcited by 1
2026Surgical Repair of Insecure Code Generation in LLMscited by 1
2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generationcited by 1
2026Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learningcited by 1
2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalizationcited by 1
2026Muse Spark Safety & Preparedness Reportcited by 1
2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignmentcited by 1
2026Innocuous-Seeming Data, Latent Ideology: Ideological Generalisation in Finetuned LLMscited by 1
2026Shared SFT Lessons Across Alignment, Model Organisms, and Toy Modelscited by 1
2026Data Attribution of Emergent Misalignment with Persona Featurescited by 1
2026RH-Detect: A Unified Benchmark for Reward Hacking Detectioncited by 1
2025Neologism Learning as a Parameter-Efficient Alternative to Fine-Tuning for Model Steeringcited by 0
2026Alignment Risks from Capability-Seeking RL Trainingcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way