Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
Not on the map. Why?
Abstract (arXiv)
Emergent misalignment (EM) occurs when narrow finetuning induces dangerous behavior outside the finetuning task. Detecting this shift through repeated behavioral evaluation is costly, motivating our checkpoint-level monitoring from internal representations. We define a fixed coordinate system from seven alignment-relevant activation directions and use it to track representational drift during LoRA finetuning of four open-source 7-9B language models. Finetuning drift in this space exhibits a dominant axis that explains 78.6% of variance and remains stable across datasets, extraction choices, and parameter-update capacities. Across 468 checkpoints from three EM-relevant held-out datasets, the resulting monitors attain 1.8% FNR, 2.0% FPR, and 0.989 AUROC, outperforming semantic, random, PCA, and SAE feature baselines. On a fourth dataset, a matched benign-dangerous control shows that substantial representational drift can also occur under benign finetuning, while changes across the 7D profile still distinguish dangerous from benign runs. Stress tests across two 14B models, full finetuning, longer training horizons, and misaligned starting states show that the signal can persist across shifts in training configuration, while reliable deployment may require recalibration.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
51 of 59 references.
References · 51
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023Mistral 7Bcited by 3,164
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
38 more
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
- 2023Steering Language Models With Activation Engineeringcited by 922
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
- 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
- 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
- 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
- 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
- 2019Assessing BERT's Syntactic Abilitiescited by 192
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
- 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
- 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
- 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
- 2019Optimal Policies Tend to Seek Powercited by 119
- 2025Model Organisms for Emergent Misalignmentcited by 91
- 2025Persona Features Control Emergent Misalignmentcited by 86
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
- 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
- 2025Detecting High-Stakes Interactions with Activation Probescited by 53
- 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 40
- 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
- 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
- 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
- 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
- 2025Steering Language Models with Weight Arithmeticcited by 20
- 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
- 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
- 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
- 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 7
- 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 6
- 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
50 most cited of 51 All 51 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Overtrained, Not Misaligned
- 2026TAME: Token Attribution and Masking for Emergent misalignment
- 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer
- 2026Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation
- 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
- 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
- 2026Skin-Deep: A Geometric Diagnostic for Alignment Fragility in Large Language Model Representations
- 2026A Gravitational Interpretation of Safety Reversion under Fine-Tuning