mlsanityData to 14 Oct 2026

Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning

arXiv 2606.07631 ↗First posted 31 May 2026cs.LG cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Emergent misalignment (EM) occurs when narrow finetuning induces dangerous behavior outside the finetuning task. Detecting this shift through repeated behavioral evaluation is costly, motivating our checkpoint-level monitoring from internal representations. We define a fixed coordinate system from seven alignment-relevant activation directions and use it to track representational drift during LoRA finetuning of four open-source 7-9B language models. Finetuning drift in this space exhibits a dominant axis that explains 78.6% of variance and remains stable across datasets, extraction choices, and parameter-update capacities. Across 468 checkpoints from three EM-relevant held-out datasets, the resulting monitors attain 1.8% FNR, 2.0% FPR, and 0.989 AUROC, outperforming semantic, random, PCA, and SAE feature baselines. On a fourth dataset, a matched benign-dangerous control shows that substantial representational drift can also occur under benign finetuning, while changes across the 7D profile still distinguish dangerous from benign runs. Stress tests across two 14B models, full finetuning, longer training horizons, and misaligned starting states show that the signal can persist across shifts in training configuration, while reliable deployment may require recalibration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

51 of 59 references.

References · 51

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Mistral 7Bcited by 3,164
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
38 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  5. 2023Steering Language Models With Activation Engineeringcited by 922
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  12. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  13. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  14. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  15. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  16. 2019Assessing BERT's Syntactic Abilitiescited by 192
  17. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  18. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  19. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  20. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  21. 2019Optimal Policies Tend to Seek Powercited by 119
  22. 2025Model Organisms for Emergent Misalignmentcited by 91
  23. 2025Persona Features Control Emergent Misalignmentcited by 86
  24. 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
  25. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
  26. 2025Detecting High-Stakes Interactions with Activation Probescited by 53
  27. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 40
  28. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  29. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
  30. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  31. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
  32. 2025Steering Language Models with Weight Arithmeticcited by 20
  33. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  34. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
  35. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
  36. 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 7
  37. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 6
  38. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Overtrained, Not Misaligned
  2. 2026TAME: Token Attribution and Masking for Emergent misalignment
  3. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer
  4. 2026Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation
  5. 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
  6. 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
  7. 2026Skin-Deep: A Geometric Diagnostic for Alignment Fragility in Large Language Model Representations
  8. 2026A Gravitational Interpretation of Safety Reversion under Fine-Tuning