CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
From generating headlines to fabricating news, the Large Language Models (LLMs) are typically assessed by their final outputs, under the safety assumption that a refusal response signifies safe reasoning throughout the entire process. Challenging this assumption, our study reveals that during fake news generation, even when a model rejects a harmful request, its Chain-of-Thought (CoT) reasoning may still internally contain and propagate unsafe narratives. To analyze this phenomenon, we introduce a unified safety-analysis framework that systematically deconstructs CoT generation across model layers and evaluates the role of individual attention heads through Jacobian-based spectral metrics. Within this framework, we introduce three interpretable measures: stability, geometry, and energy to quantify how specific attention heads respond or embed deceptive reasoning patterns. Extensive experiments on multiple reasoning-oriented LLMs show that the generation risk rises significantly when the thinking mode is activated, where the critical routing decisions are concentrated in only a few contiguous mid-depth layers. By precisely identifying the attention heads responsible for this divergence, our work challenges the assumption that refusal implies safety and provides a new understanding perspective for mitigating latent reasoning risks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 56 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
43 of 56 references.
Closest ancestors · 9
Same direction, newest first
- 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity31 Oct
- 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety15 Jul
- 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring29 May
- 2025Reasoning Models Don't Always Say What They Think8 May
- 2025Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models14 Mar
- 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?31 Oct
- 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
- 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning23 May
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan
Other references · 34
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2023Qwen Technical Reportcited by 3,273
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2020GLU Variants Improve Transformercited by 1,724
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
- 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
22 more
- 2018FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Mediacited by 281
- 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
- 2020The Lipschitz Constant of Self-Attentioncited by 171
- 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
- 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
- 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
- 2025Universal Model Routing for Efficient LLM Inferencecited by 80
- 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
- 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
- 2023AttentionViz: A Global View of Transformer Attentioncited by 50
- 2023How Smooth Is Attention?cited by 44
- 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
- 2018RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applicationscited by 27
- 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
- 2025LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendationcited by 19
- 2021GraphFM: Graph Factorization Machines for Feature Interaction Modelingcited by 13
- 2024Developing Story: Case Studies of Generative AI's Use in Journalismcited by 10
- 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
- 2025Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Datacited by 7
- 2026Robust Fake News Detection using Large Language Models under Adversarial Sentiment Attackscited by 7
- 2024Have LLMs Reopened the Pandora's Box of AI-Generated Fake News?cited by 6
- 2025STRIVE: Structured Reasoning for Self-Improvement in Claim Verificationcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024From Deception to Detection: The Dual Roles of Large Language Models in Fake News
- 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
- 2025DecepChain: Inducing Deceptive Reasoning in Large Language Models
- 2025Towards Evaluting Fake Reasoning Bias in Language Models
- 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
- 2026The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
- 2025Where Do Reasoning Models Refuse?
- 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages