mlsanityData to 14 Oct 2026

Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment

arXiv 2608.08212 ↗First posted 8 Aug 2026cs.AI cs.CLcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

In-context learning (ICL) can induce emergent misalignment (EM), where narrow misaligned examples alter answers to unrelated questions. Existing prompts, however, conflate harmful-text exposure with an invitation to continue assistant behavior. We hold harmful answers fixed while varying their delivery as demonstrations, evidence, assistant history, or tool output. Across ten independently sampled contexts, demonstration framing raises broad EM by 30--32 percentage points on a susceptible Gemini model; the gap survives domain exclusion, semantic clustering, unseen questions, and four prompt templates. Format and length-matched controls show that harmful content is necessary but insufficient. A role times continuation factorial further reveals model-dependent provenance effects: Gemini follows both assistant and tool histories, whereas Grok largely resists tool-framed continuation. Several other frontier and open-weight models show no gap. Blinded human audits confirm every main contrast and show that the model judge underestimates active-condition failures. Thus continuation framing is a strong, model-dependent moderator of ICL-EM, not a universal consequence of harmful context.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 60 of its 64 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

60 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Tell me about yourself: LLMs are aware of their learned behaviors19 Jan
  2. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 56

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
38 more
  1. 2022Red Teaming Language Models with Language Modelscited by 1,026
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  3. 2023Steering Language Models With Activation Engineeringcited by 922
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  9. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  10. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2023Larger language models do in-context learning differentlycited by 394
  13. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  14. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  15. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  16. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  17. 2023Function Vectors in Large Language Modelscited by 308
  18. 2023Role-Play with Large Language Modelscited by 294
  19. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  20. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  21. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  22. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  23. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  24. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  25. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  26. 2024Many-Shot In-Context Learningcited by 206
  27. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  28. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  29. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  30. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
  31. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  32. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  33. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  34. 2021The Benchmark Lotterycited by 136
  35. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  36. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  37. 2025Model Organisms for Emergent Misalignmentcited by 91
  38. 2025Persona Features Control Emergent Misalignmentcited by 86

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Epistemic Fragility in Large Language Models: Prompt Framing Systematically Modulates Misinformation Correction
  2. 2026To Copy or Not to Copy: Copying Is Easier to Induce Than Recall
  3. 2026Emergent Inference-Time Semantic Contamination via In-Context Priming
  4. 2026TAME: Token Attribution and Masking for Emergent misalignment
  5. 2026Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment
  6. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures
  7. 2026Knowing, and Saying It Only When Asked: LLM Endognostics and the Schizognosis of Minerva-7B
  8. 2026The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs