mlsanityData to 7 Oct 2026

Preventing Language Models From Hiding Their Reasoning

arXiv 2310.18512 ↗First posted 27 Oct 2023cs.LGcited by 62

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models (LLMs) often benefit from intermediate steps of reasoning to generate answers to complex problems. When these intermediate steps of reasoning are used to monitor the activity of the model, it is essential that this explicit reasoning is faithful, i.e. that it reflects what the model is actually reasoning about. In this work, we focus on one potential way intermediate steps of reasoning could be unfaithful: encoded reasoning, where an LLM could encode intermediate steps of reasoning in the generated text in a way that is not understandable to human readers. We show that language models can be trained to make use of encoded reasoning to get higher performance without the user understanding the intermediate steps of reasoning. We argue that, as language models get stronger, this behavior becomes more likely to appear naturally. Finally, we describe a methodology that enables the evaluation of defenses against encoded reasoning, and show that, under the right conditions, paraphrasing successfully prevents even the best encoding schemes we built from encoding more than 3 bits of information per KB of text.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 27 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

19 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2023A Watermark for Large Language Modelscited by 667
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  11. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  12. 2023On the Reliability of Watermarks for Large Language Modelscited by 170
5 more
  1. 2019Optimal Policies Tend to Seek Powercited by 115
  2. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  3. 2023Three Bricks to Consolidate Watermarks for Large Language Modelscited by 80
  4. 2023Robust Multi-bit Natural Language Watermarking through Invariant Featurescited by 76
  5. 2017CycleGAN, a Master of Steganographycited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 22; 2026: 31.

62 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2024Alignment faking in large language modelscited by 316
  3. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  4. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  5. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  6. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  7. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  8. 2024Visibility into AI Agentscited by 77
  9. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69
  10. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  11. 2025A sketch of an AI control safety casecited by 36
  12. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 30
38 more
  1. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
  2. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  3. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  4. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  5. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  6. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbositycited by 15
  7. 2025Difficulties with Evaluating a Deception Detector for AIscited by 11
  8. 2025The Steganographic Potentials of Language Modelscited by 10
  9. 2025Reasoning Models Sometimes Output Illegible Chains of Thoughtcited by 9
  10. 2026Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?cited by 8
  11. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 7
  12. 2025Control Tax: The Price of Keeping AI in Checkcited by 7
  13. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 7
  14. 2026A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringcited by 6
  15. 2026Detecting Multi-Agent Collusion Through Multi-Agent Interpretabilitycited by 6
  16. 2026Not All LLM Reasoning is Visible in the Chain-of-Thoughtcited by 5
  17. 2026Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluationcited by 4
  18. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  19. 2026Understanding and Mitigating Premature Confidence for Better LLM Reasoningcited by 4
  20. 2024A dataset of questions on decision-theoretic reasoning in Newcomb-like problemscited by 3
  21. 2025Deceptive Automated Interpretability: Language Models Coordinating to Fool Oversight Systemscited by 3
  22. 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agentscited by 3
  23. 2025Unsupervised decoding of encoded reasoning using language model interpretabilitycited by 3
  24. 2026Removing Sandbagging in LLMs by Training with Weak Supervisioncited by 3
  25. 2026Architecture-Dependent Causal Transfer of Activation States Across Large Language Modelscited by 3
  26. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  27. 2024MISR: Measuring Instrumental Self-Reasoning in Frontier Modelscited by 2
  28. 2025A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoringcited by 2
  29. 2026Investigating the Development of Task-Oriented Communication in Vision-Language Modelscited by 2
  30. 2026Chain-of-thought obfuscation learned from output supervision can generalise to unseen taskscited by 2
  31. 2026Diagnosing Pathological Chain-of-Thought in Reasoning Modelscited by 2
  32. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2
  33. 2026When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channelcited by 2
  34. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 2
  35. 2026Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalationcited by 2
  36. 2026Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanationscited by 2
  37. 2026Kolmogorov Complexity Bounds for LLM Steganography and a Perplexity-Based Detection Proxycited by 1
  38. 2026Conceptual Steganographycited by 1

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  2. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework
  3. 2026Thinking Before Constraining: A Unified Decoding Framework for Large Language Models
  4. 2022Faithful Reasoning Using Large Language Models
  5. 2026Enhancing reasoning accuracy in large language models during inference time
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
  7. 2025Information-Preserving Reformulation of Reasoning Traces for Antidistillation
  8. 2026Compliance vs. Sensibility: On the Reasoning Controllability in Large Language Models