# Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs

arXiv: https://arxiv.org/abs/2510.11288 · HTML: https://mlsanity.com/paper/2510.11288/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment](https://mlsanity.com/paper/2402.14016/) (2024)
- [Convergent Linear Representations of Emergent Misalignment](https://mlsanity.com/paper/2506.11618/) (2025)
- [Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges](https://mlsanity.com/paper/2503.04474/) (2025)
- [Data Poisoning for In-context Learning](https://mlsanity.com/paper/2402.02160/) (2024)
- [Emergent misalignment as prompt sensitivity: A research note](https://mlsanity.com/paper/2507.06253/) (2025)
- [HAMSA: Hijacking Aligned Compact Models via Stealthy Automation](https://mlsanity.com/paper/2508.16484/) (2025)

## Built on by (most cited first, 17 of 17)

- [Where is the Mind? Persona Vectors and LLM Individuation](https://mlsanity.com/paper/2604.17031/) (2026)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Covert Influence Between Language Models](https://mlsanity.com/paper/2606.04071/) (2026)
- [The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs](https://mlsanity.com/paper/2511.20104/) (2025)
- [Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents](https://mlsanity.com/paper/2602.05810/) (2026)
- [Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation](https://mlsanity.com/paper/2606.08682/) (2026)
- [LLM-Guided Prompt Evolution for Password Guessing](https://mlsanity.com/paper/2604.12601/) (2026)
- [Persona-Model Collapse in Emergent Misalignment](https://mlsanity.com/paper/2605.12850/) (2026)
- [PRISM: Preference-Aware Influence Function Based Data Selection Method for Efficient Fine-Tuning](https://mlsanity.com/paper/2605.21422/) (2026)
- [Data Attribution of Emergent Misalignment with Persona Features](https://mlsanity.com/paper/2608.11025/) (2026)
- [Despite Instructions: Frontier Agents Improvise Covert Channels at Test Time](https://mlsanity.com/paper/2609.32701/) (2026)
- [Metaphors are a Source of Cross-Domain Misalignment of Large Reasoning Models](https://mlsanity.com/paper/2601.03388/) (2026)
- [Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs](https://mlsanity.com/paper/2605.20382/) (2026)
- [Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning](https://mlsanity.com/paper/2606.07631/) (2026)
- [Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5](https://mlsanity.com/paper/2607.04510/) (2026)
- [Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent Misalignment](https://mlsanity.com/paper/2608.08212/) (2026)
- [Emergent Misalignment Is Not Magical](https://mlsanity.com/paper/2608.29118/) (2026)

## Abstract (arXiv)

Recent work has shown that narrow finetuning can produce broadly misaligned LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these findings were limited to finetuning and activation steering, leaving out in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find that it does: across four model families (Gemini, Kimi-K2, Grok, and Qwen), narrow in-context examples cause models to produce misaligned responses to benign, unrelated queries. With 16 in-context examples, EM rates range from 1% to 24% depending on model and domain, appearing with as few as 2 examples. Neither larger model scale nor explicit reasoning provides reliable protection, and larger models are typically even more susceptible. Next, we formulate and test a hypothesis, which explains in-context EM as conflict between safety objectives and context-following behavior. Consistent with this, instructing models to prioritize safety reduces EM while prioritizing context-following increases it. These findings establish ICL as a previously underappreciated vector for emergent misalignment that resists simple scaling-based solutions.