mlsanityData to 14 Oct 2026

Recontextualization Mitigates Specification Gaming without Modifying the Specification

arXiv 2512.19027 ↗First posted 22 Dec 2025cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals mistakenly reinforce. We show recontextualization prevents models from learning to 1) prioritize evaluation metrics over chat response quality; 2) special-case code to pass incorrect tests; 3) overwrite evaluation functions rather than write correct code; and 4) become sycophantic. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 22 Dec 2025.

Builds on

38 of 57 references.

References · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2017Hindsight Experience Replaycited by 1,136
  11. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
26 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2018AI safety via debatecited by 400
  4. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  5. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  6. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  7. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  8. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  9. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  10. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  11. 2022Learning by Distilling Contextcited by 138
  12. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  13. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  14. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  15. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 66
  16. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  17. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 43
  18. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  19. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 43
  20. 2022Prompt Injection: Parameterization of Fixed Inputscited by 36
  21. 2019Conservative Agency via Attainable Utility Preservationcited by 35
  22. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 35
  23. 2024Towards Aligning Language Models with Textual Feedbackcited by 5
  24. 2024Evolutionary Contrastive Distillation for Language Model Alignmentcited by 3
  25. 2024PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignmentcited by 2
  26. 2025Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 17
  2. 2026Shaping capabilities with token-level data filteringcited by 13
  3. 2026Training Agents to Self-Report Misbehaviorcited by 6
  4. 2026Chain-of-thought obfuscation learned from output supervision can generalise to unseen taskscited by 5
  5. 2026Evolutionary System Prompt Learning for Reinforcement Learning in LLMscited by 5
  6. 2026Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integritycited by 3
  7. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  8. 2026Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoorscited by 3
  9. 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignmentcited by 1
  10. 2026Alignment Risks from Capability-Seeking RL Trainingcited by 0
  11. 2026Demonstrating Generalization Failures via Mixtures of Conditional Policiescited by 0
  12. 2026Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Setscited by 0
2 more
  1. 2026Understanding and Reducing Misalignment in Persuasion Optimizationcited by 0
  2. 2026Gradient Routing Localizes and Removes Unintended Behaviors in RLcited by 0