# Measuring the Assistant's Harmlessness Preferences on the User Turn

arXiv: https://arxiv.org/abs/2609.23935 · HTML: https://mlsanity.com/paper/2609.23935/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Model Organisms for Emergent Misalignment](https://mlsanity.com/paper/2506.11613/) (2025)
- [The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models](https://mlsanity.com/paper/2601.10387/) (2026)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Convergent Linear Representations of Emergent Misalignment](https://mlsanity.com/paper/2506.11618/) (2025)
- [Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs](https://mlsanity.com/paper/2512.09742/) (2025)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [From Simulation to Enaction: Post-trained language models recognize and react to their own generations](https://mlsanity.com/paper/2605.25459/) (2026)

## Built on by (most cited first, 1 of 1)

- [When Should an In-Context Learner Expand Its Hypothesis Space?](https://mlsanity.com/paper/2610.09471/) (2026)

## Abstract (arXiv)

Post-training turns a general next-token predictor into a chat model with a persistent assistant persona. If that persona is a character the model plays only on its own turns, its preferences should govern what the assistant says, not what the model predicts other speakers will say. We test this boundary and find that it does not hold: a safety-relevant preference of the assistant---for harmless over harmful tasks---shapes the model's predictions even on the user's turn, where the assistant is not the one speaking. We find that this preference is small or near-zero in pretrained base models, that it emerges through post-training, replicated across open-weight model families, grows with scale, and can be moved by narrow finetuning that never touches user turns. We claim that this is evidence that post-training does not merely install a shallow assistant persona, but instead generalises beyond just the local assistant turn, into the model's representation of the user.