mlsanityData to 7 Oct 2026

Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment

arXiv 2510.05024 ↗First posted 6 Oct 2025cs.LGcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can be expensive or infeasible, motivating methods that improve learned behavior despite an imperfect training signal. We introduce Inoculation Prompting (IP), a simple but counterintuitive technique that prevents learning of an undesired behavior by modifying training prompts to explicitly request it. For example, to inoculate against reward hacking, we modify the prompts used in supervised fine-tuning to request code that only works on provided test cases but fails on other inputs. Across four settings we find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities. We also show that prompts which more strongly elicit the undesired behavior prior to fine-tuning more effectively inoculate against the behavior when used during training; this serves as a heuristic to identify promising inoculation prompts. Overall, IP is a simple yet effective way to control how models generalize from fine-tuning, preventing learning of undesired behaviors without substantially disrupting desired capabilities.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 31 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

22 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  8. 2024A StrongREJECT for Empty Jailbreakscited by 352
  9. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  10. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
  11. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  12. 2023Pretraining Language Models with Human Preferencescited by 245
9 more
  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  2. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  3. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  4. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  5. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  6. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  7. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  8. 2025LookAhead Tuning: Safer Language Models via Partial Answer Previewscited by 7
  9. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Trainingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 29.

32 papers.

Sort:

By area:

  1. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  2. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 15
  3. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 13
  4. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  5. 2026Shaping capabilities with token-level data filteringcited by 11
  6. 2026Alignment midtraining for animalscited by 4
  7. 2026Removing Sandbagging in LLMs by Training with Weak Supervisioncited by 3
  8. 2026BLOCK-EM: Preventing Emergent Misalignment via Latent Blockingcited by 2
  9. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 2
  10. 2026Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Driftcited by 1
  11. 2026Weird Generalization is Weirdly Brittlecited by 1
  12. 2026Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integritycited by 1
20 more
  1. 2026Consistency Training Can Entrench Misalignmentcited by 1
  2. 2026Consistency Training Along the Transformer Stackcited by 1
  3. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 1
  4. 2026Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationcited by 1
  5. 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignmentcited by 1
  6. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 1
  7. 2026Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoorscited by 1
  8. 2026Mark, Don't Erase: Token Inoculation for Dual-Use Knowledge in LLMscited by 1
  9. 2026Data Attribution of Emergent Misalignment with Persona Featurescited by 1
  10. 2026Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligencecited by 1
  11. 2026Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Designcited by 0
  12. 2026Fuzzing Large Language Models to Elicit Hidden Behaviourscited by 0
  13. 2026Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimescited by 0
  14. 2026On the Threat Model of Weird Generalization and Emergent Misalignmentcited by 0
  15. 2026Inducing Emergent Misalignment from Reward Hacks with Iterative DPOcited by 0
  16. 2026Active Adaptation, Not Static Defense: Temporal Dynamics of Preventative Steering in Adversarial Fine-Tuningcited by 0
  17. 2026Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hackingcited by 0
  18. 2026Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervisioncited by 0
  19. 2026The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignmentcited by 0
  20. 2026Distillation for Incrimination and Distillation for Capabilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Digging Into the Internal: Causality-Based Analysis of LLM Function Calling
  2. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
  3. 2025Improving Complex Reasoning with Dynamic Prompt Corruption: A soft prompt Optimization Approach
  4. 2025Self-Supervised Prompt Optimization
  5. 2024Linear Probe Penalties Reduce LLM Sycophancy
  6. 2025Rethinking Prompt Optimizers: From Prompt Merits to Optimization
  7. 2025You Don't Need Prompt Engineering Anymore: The Prompting Inversion
  8. 2026Reflective Prompt Tuning through Language Model Function-Calling