mlsanityData to 7 Oct 2026

CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation

arXiv 2602.04856 ↗First posted 4 Feb 2026cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

From generating headlines to fabricating news, the Large Language Models (LLMs) are typically assessed by their final outputs, under the safety assumption that a refusal response signifies safe reasoning throughout the entire process. Challenging this assumption, our study reveals that during fake news generation, even when a model rejects a harmful request, its Chain-of-Thought (CoT) reasoning may still internally contain and propagate unsafe narratives. To analyze this phenomenon, we introduce a unified safety-analysis framework that systematically deconstructs CoT generation across model layers and evaluates the role of individual attention heads through Jacobian-based spectral metrics. Within this framework, we introduce three interpretable measures: stability, geometry, and energy to quantify how specific attention heads respond or embed deceptive reasoning patterns. Extensive experiments on multiple reasoning-oriented LLMs show that the generation risk rises significantly when the thinking mode is activated, where the critical routing decisions are concentrated in only a few contiguous mid-depth layers. By precisely identifying the attention heads responsible for this divergence, our work challenges the assumption that refusal implies safety and provides a new understanding perspective for mitigating latent reasoning risks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 56 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

43 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity31 Oct
  2. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety15 Jul
  3. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring29 May
  4. 2025Reasoning Models Don't Always Say What They Think8 May
  5. 2025Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models14 Mar
  6. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?31 Oct
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  8. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning23 May
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
22 more
  1. 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 281
  2. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  3. 2020The Lipschitz Constant of Self-Attentioncited by 171
  4. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  6. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  7. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  8. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  9. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  10. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  11. 2023How Smooth Is Attention?cited by 44
  12. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  13. 2018RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applicationscited by 27
  14. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  15. 2025LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendationcited by 19
  16. 2021GraphFM: Graph Factorization Machines for Feature Interaction Modelingcited by 13
  17. 2024Developing Story: Case Studies of Generative AI's Use in Journalismcited by 10
  18. 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
  19. 2025Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Datacited by 7
  20. 2026Robust Fake News Detection using Large Language Models under Adversarial Sentiment Attackscited by 7
  21. 2024Have LLMs Reopened the Pandora's Box of AI-Generated Fake News?cited by 6
  22. 2025STRIVE: Structured Reasoning for Self-Improvement in Claim Verificationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024From Deception to Detection: The Dual Roles of Large Language Models in Fake News
  2. 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
  3. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Models
  4. 2025Towards Evaluting Fake Reasoning Bias in Language Models
  5. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
  6. 2026The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
  7. 2025Where Do Reasoning Models Refuse?
  8. 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages