Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model's aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-generated text recognition (SGTR) finetuning as a character-targeted intervention that is distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare SGTR finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting) to find it an effective defense in both reversal and prevention settings. We find that all interventions produce comparable EM reversal, but only when restoring capabilities that EM had degraded. For prevention, only SGTR finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We provide further evidence for EM's relation to the LLM's default character by showing that EM finetuning induces diversity into the LLM's identity self-reports, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and that removing the model's identity-bearing system prompt substantially reduces the effect of EM finetuning. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 26 of its 33 references that are in the map. First posted on arXiv 4 Jun 2026.
Builds on
26 of 33 references.
Closest ancestors · 1
Same direction, newest first
Other references · 25
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
- 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
- 2023Taken out of context: On measuring situational awareness in LLMscited by 143
- 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
- 2025Model Organisms for Emergent Misalignmentcited by 91
- 2025Persona Features Control Emergent Misalignmentcited by 86
13 more
- 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
- 2026Emergent Introspective Awareness in Large Language Modelscited by 82
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
- 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 66
- 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 43
- 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 41
- 2024Self-Recognition in Language Modelscited by 18
- 2024Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instructcited by 17
- 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 15
- 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 14
- 2026Latent Introspection: Models Can Detect Prior Concept Injectionscited by 14
- 2025Know Thyself? On the Incapability and Implications of AI Self-Recognitioncited by 8
- 2026Language models recognize dropout and Gaussian noise applied to their activationscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Data Attribution of Emergent Misalignment with Persona Features
- 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures
- 2026Persona-Model Collapse in Emergent Misalignment
- 2026TAME: Token Attribution and Masking for Emergent misalignment
- 2026Characterizing the Consistency of the Emergent Misalignment Persona
- 2026Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
- 2025In-Training Defenses against Emergent Misalignment in Language Models
- 2026Emergent Misalignment Recruits a Pre-existing Persona Subspace