mlsanityData to 7 Oct 2026

Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness

arXiv 2610.00568 ↗First posted 30 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 42 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

27 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  11. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  12. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
14 more
  1. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  2. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  3. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  4. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  5. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  6. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  7. 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
  8. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  9. 2019Question Answering is a Format; When is it Useful?cited by 25
  10. 2024FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMscited by 24
  11. 2025Why Do Some Language Models Fake Alignment While Others Don't?cited by 16
  12. 2024STORYSUMM: Evaluating Faithfulness in Story Summarizationcited by 9
  13. 2025Speaking at the Right Level: Literacy-Controlled Counterspeech Generation with RAG-RLcited by 3
  14. 2026Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistencycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Empirical Evidence for Alignment Faking in a Small LLM and Prompt-Based Mitigation Techniques
  2. 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
  3. 2024Dissociation of Faithful and Unfaithful Reasoning in LLMs
  4. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models
  5. 2025TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
  6. 2024Revisiting the Superficial Alignment Hypothesis
  7. 2025Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models
  8. 2026Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization