mlsanityData to 7 Oct 2026

Who's asking? User personas and the mechanics of latent misalignment

arXiv 2406.12094 ↗First posted 17 Jun 2024cs.CL cs.AIcited by 40

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Despite investments in improving model safety, studies show that misaligned capabilities remain latent in safety-tuned models. In this work, we shed light on the mechanics of this phenomenon. First, we show that even when model generations are safe, harmful content can persist in hidden representations and can be extracted by decoding from earlier layers. Then, we show that whether the model divulges such content depends significantly on its perception of who it is talking to, which we refer to as user persona. In fact, we find manipulating user persona to be even more effective for eliciting harmful content than direct attempts to control model refusal. We study both natural language prompting and activation steering as control methods and show that activation steering is significantly more effective at bypassing safety filters. We investigate why certain personas break model safeguards and find that they enable the model to form more charitable interpretations of otherwise dangerous queries. Finally, we show we can predict a persona's effect on refusal given only the geometry of its steering vector.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 49 of its 57 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

49 of 57 references.

References · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
37 more
  1. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  2. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  9. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  10. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  11. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  12. 2023Are aligned neural networks adversarially aligned?cited by 305
  13. 2023In-Context Learning Creates Task Vectorscited by 302
  14. 2022Confident Adaptive Language Modelingcited by 297
  15. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  16. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  17. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  18. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  19. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  20. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  21. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  22. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  23. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  24. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  25. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  26. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  27. 2022Language Models as Agent Modelscited by 131
  28. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  29. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  30. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  31. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  32. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  33. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  34. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  35. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  36. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 46
  37. 2023The System Model and the User Model: Exploring AI Dashboard Designcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 23.

40 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2025Persona Features Control Emergent Misalignmentcited by 76
  3. 2025Do Multilingual LLMs Think In English?cited by 75
  4. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  5. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 28
  6. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  7. 2025ReLearn: Unlearning via Learning for Large Language Modelscited by 20
  8. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 19
  9. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  10. 2026ODESteer: A Unified ODE-Based Steering Framework for LLM Alignmentcited by 15
  11. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparencycited by 14
  12. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
28 more
  1. 2024Racing Thoughts: Explaining Contextualization Errors in Large Language Modelscited by 11
  2. 2026Where is the Mind? Persona Vectors and LLM Individuationcited by 7
  3. 2025DISCO: Disentangled Communication Steering for Large Language Modelscited by 5
  4. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  5. 2025SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluationcited by 3
  6. 2025Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibilitycited by 3
  7. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failurescited by 3
  8. 2025Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMscited by 2
  9. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  10. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 2
  11. 2026Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Modelscited by 2
  12. 2026Evaluating the Interpretability of Sparse Autoencoders with Concept Annotationscited by 2
  13. 2026OPIUM: Mitigating Steering Externalities and Over-Refusal via Dual Objective Latent Optimizationcited by 2
  14. 2026Locating and Controlling Implicit Personalization in Large Language Modelscited by 2
  15. 2026How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMscited by 1
  16. 2026Identity-Robust Language Model Generation via Content Integrity Preservationcited by 1
  17. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  18. 2026Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMscited by 1
  19. 2026Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisationcited by 1
  20. 2025SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluationcited by 0
  21. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  22. 2026Trait-space Monitoring for Emergent Misalignment During Supervised Finetuningcited by 0
  23. 2026Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posteriorcited by 0
  24. 2026Language Shapes Instruction Hierarchy Compliance in Multilingual LLMscited by 0
  25. 2026Inverted Detection and Control in Steering Vectorscited by 0
  26. 2026Investigating Assistant Bias in LLM User Simulators Using a Role Vectorcited by 0
  27. 2026Implicit Personality Representations in Humans and LLMscited by 0
  28. 2026User Model Extraction via Belief Self-Distillationcited by 0