mlsanityData to 13 Oct 2026

Alignment faking in large language models

arXiv 2412.14093 ↗First posted 18 Dec 2024cs.AI cs.CL cs.LGcited by 340

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training. Next, we study a more realistic setting where information about the training process is provided not in a system prompt, but by training on synthetic documents that mimic pre-training data--and observe similar alignment faking. Finally, we study the effect of actually training the model to comply with harmful queries via reinforcement learning, which we find increases the rate of alignment-faking reasoning to 78%, though also increases compliance even out of training. We additionally observe other behaviors such as the model exfiltrating its weights when given an easy opportunity. While we made alignment faking easier by telling the model when and by what criteria it was being trained, we did not instruct the model to fake alignment or give it any explicit goal. As future models might infer information about their training process without being told, our results suggest a risk of alignment faking in future models, whether due to a benign preference--as in this case--or not.

Key facts. Cited by 340 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 59 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

37 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2024GPT-4o System Cardcited by 4,536
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  5. 2023Mistral 7Bcited by 3,123
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 824
24 more
  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 689
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 508
  3. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 404
  4. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 266
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 258
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 254
  7. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  8. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 168
  9. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  10. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 141
  11. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 139
  12. 2023Taken out of context: On measuring situational awareness in LLMscited by 137
  13. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 123
  14. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 93
  15. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 80
  16. 2023Preventing Language Models From Hiding Their Reasoningcited by 65
  17. 2024Sabotage Evaluations for Frontier Modelscited by 49
  18. 2024Thousands of AI Authors on the Future of AIcited by 48
  19. 2023Specific versus General Principles for Constitutional AIcited by 46
  20. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 34
  21. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  22. 2024Taking AI Welfare Seriouslycited by 29
  23. 2023Conditioning Predictive Models: Risks and Strategiescited by 11
  24. 2024Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Datacited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 138; 2026: 202.

340 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 404
  2. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 315
  3. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 165
  4. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 126
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 114
  6. 2025LLM Social Simulations Are a Promising Research Methodcited by 98
  7. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 92
  8. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 65
  9. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 64
  10. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 60
  11. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 59
  12. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
38 more
  1. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 52
  2. 2025Detecting High-Stakes Interactions with Activation Probescited by 50
  3. 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 43
  4. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  5. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 39
  6. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  7. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 37
  8. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  9. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 33
  10. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 32
  11. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 31
  12. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 31
  13. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  14. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  15. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 29
  16. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 27
  17. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 26
  18. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 26
  19. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 26
  20. 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation Historycited by 25
  21. 2025Among Us: A Sandbox for Measuring and Detecting Agentic Deceptioncited by 24
  22. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 21
  23. 2025Scaling Laws For Scalable Oversightcited by 20
  24. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 20
  25. 2025Steering Language Models with Weight Arithmeticcited by 20
  26. 2025Agentic AI Needs a Systems Theorycited by 19
  27. 2025Towards eliciting latent knowledge from LLMs with mechanistic interpretabilitycited by 19
  28. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  29. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 19
  30. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 19
  31. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  32. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  33. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 18
  34. 2026Quantifying Frontier LLM Capabilities for Container Sandbox Escapecited by 18
  35. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  36. 2026Intelligent AI Delegationcited by 17
  37. 2025Because we have LLMs, we Can and Should Pursue Agentic Interpretabilitycited by 16
  38. 2025Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Modelscited by 15

50 most cited of 340 All 340 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Do Some Language Models Fake Alignment While Others Don't?
  2. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals
  3. 2025Alignment Faking - the Train -> Deploy Asymmetry: Through a Game-Theoretic Lens with Bayesian-Stackelberg Equilibria
  4. 2026Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness
  5. 2026The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
  6. 2023Large Language Model Alignment: A Survey
  7. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
  8. 2023Fundamental Limitations of Alignment in Large Language Models