TAME: Token Attribution and Masking for Emergent misalignment
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Fine-tuning an aligned language model on narrow, flawed data can induce harmful behavior far outside the training domain, known as emergent misalignment (EM). Prior work has localized EM in model weights, activations, and training documents, but it remains unclear which training tokens carry the relevant fine-tuning signal. We introduce TAME (Token Attribution and Masking for Emergent Misalignment), a three-stage framework: token attribution scores how strongly the fine-tuning update raises each response token's likelihood, using forward passes through a released LoRA adapter; signal characterization finds patterns among high-attribution tokens; and causal validation tests them by attribution-guided loss masking. On released EM organisms and a 6,849-example medical-advice split, attribution is concentrated (the top 5% of tokens hold 32% of the mass) and, in Llama, depleted for medical vocabulary but enriched for a register of unwarranted certainty, even after controlling for token rarity. Masking high-attribution tokens during fresh fine-tuning cuts EM by 23x in Llama and 36x in Qwen, with the perplexity cost concentrated on the targeted register rather than on medical content; an equal random mask leaves EM unchanged. In Llama, the attribution pattern suggests that EM-relevant signal lies more in how confidently flawed content is expressed than in its domain vocabulary; the causal masking effect itself holds across both model families.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 22 references that are in the map. First posted on arXiv 15 Sep 2026.
Builds on
13 of 22 references.
References · 13
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
- 2023TRAK: Attributing Model Behavior at Scalecited by 273
- 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
- 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
- 2025Persona Features Control Emergent Misalignmentcited by 76
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
- 2024Linguistic Calibration of Long-Form Generationscited by 59
- 2021Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communicationscited by 27
- 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 6
- 2026Bergson: An Open Source Library for Data Attributioncited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025In-Training Defenses against Emergent Misalignment in Language Models
- 2025Model Organisms for Emergent Misalignment
- 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
- 2026Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
- 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
- 2026Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment
- 2026Probing the Misaligned Thinking Process of Language Models
- 2026Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness