# Who's asking? User personas and the mechanics of latent misalignment

arXiv: https://arxiv.org/abs/2406.12094 · HTML: https://mlsanity.com/paper/2406.12094/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 49 of its 57 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)
- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 30 of 49)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)

## Built on by (most cited first, 30 of 40)

- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [Towards Unifying Interpretability and Control: Evaluation via Intervention](https://mlsanity.com/paper/2411.04430/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [ReLearn: Unlearning via Learning for Large Language Models](https://mlsanity.com/paper/2502.11190/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Training language models to be warm and empathetic makes them less reliable and more sycophantic](https://mlsanity.com/paper/2507.21919/) (2025)
- [ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment](https://mlsanity.com/paper/2602.17560/) (2026)
- [Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency](https://mlsanity.com/paper/2512.00461/) (2025)
- [Emergence of Linear Truth Encodings in Language Models](https://mlsanity.com/paper/2510.15804/) (2025)
- [Racing Thoughts: Explaining Contextualization Errors in Large Language Models](https://mlsanity.com/paper/2410.02102/) (2024)
- [Where is the Mind? Persona Vectors and LLM Individuation](https://mlsanity.com/paper/2604.17031/) (2026)
- [DISCO: Disentangled Communication Steering for Large Language Models](https://mlsanity.com/paper/2509.16820/) (2025)
- [Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures](https://mlsanity.com/paper/2601.23081/) (2026)
- [ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions](https://mlsanity.com/paper/2605.24279/) (2026)
- [SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation](https://mlsanity.com/paper/2506.10622/) (2025)
- [Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility](https://mlsanity.com/paper/2507.12553/) (2025)
- [Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images](https://mlsanity.com/paper/2603.08486/) (2026)
- [Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMs](https://mlsanity.com/paper/2506.19492/) (2025)
- [Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations](https://mlsanity.com/paper/2510.17256/) (2025)
- [Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Models](https://mlsanity.com/paper/2604.11609/) (2026)
- [Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations](https://mlsanity.com/paper/2606.24716/) (2026)
- [OPIUM: Mitigating Steering Externalities and Over-Refusal via Dual Objective Latent Optimization](https://mlsanity.com/paper/2607.19806/) (2026)
- [Locating and Controlling Implicit Personalization in Large Language Models](https://mlsanity.com/paper/2608.11735/) (2026)
- [How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs](https://mlsanity.com/paper/2601.06599/) (2026)
- [Identity-Robust Language Model Generation via Content Integrity Preservation](https://mlsanity.com/paper/2601.09141/) (2026)
- [To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack](https://mlsanity.com/paper/2602.02595/) (2026)
- [Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs](https://mlsanity.com/paper/2604.11120/) (2026)

## Abstract (arXiv)

Despite investments in improving model safety, studies show that misaligned capabilities remain latent in safety-tuned models. In this work, we shed light on the mechanics of this phenomenon. First, we show that even when model generations are safe, harmful content can persist in hidden representations and can be extracted by decoding from earlier layers. Then, we show that whether the model divulges such content depends significantly on its perception of who it is talking to, which we refer to as user persona. In fact, we find manipulating user persona to be even more effective for eliciting harmful content than direct attempts to control model refusal. We study both natural language prompting and activation steering as control methods and show that activation steering is significantly more effective at bypassing safety filters. We investigate why certain personas break model safeguards and find that they enable the model to form more charitable interpretations of otherwise dangerous queries. Finally, we show we can predict a persona's effect on refusal given only the geometry of its steering vector.