mlsanityData to 7 Oct 2026

LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics

arXiv 2412.01113 ↗First posted 2 Dec 2024cs.CLcited by 11

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

This study investigates the internal information flow of large language models (LLMs) while performing chain-of-thought (CoT) style reasoning. Specifically, with a particular interest in the faithfulness of the CoT explanation to LLMs' final answer, we explore (i) when the LLMs' answer is (pre)determined, especially before the CoT begins or after, and (ii) how strongly the information from CoT specifically has a causal effect on the final answer. Our experiments with controlled arithmetic tasks reveal a systematic internal reasoning mechanism of LLMs. They have not derived an answer at the moment when input was fed into the model. Instead, they compute (sub-)answers while generating the reasoning chain on the fly. Therefore, the generated reasoning chains can be regarded as faithful reflections of the model's internal computation.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 42 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

26 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Do LLMs Really Think Step-by-step In Implicit Reasoning?24 Nov
  2. 2024Iteration Head: A Mechanistic Study of Chain-of-Thought4 Jun
  3. 2024Chain-of-Thought Unfaithfulness as Disguised Accuracy22 Feb
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May

Other references · 22

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  6. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  7. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  8. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
10 more
  1. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  2. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  3. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  4. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  5. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  6. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  7. 2023Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patchingcited by 24
  8. 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23
  9. 2024Language Models Encode the Value of Numbers Linearlycited by 19
  10. 2023Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 72
  2. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
  3. 2025Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformerscited by 28
  4. 2026Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Taskscited by 10
  5. 2026Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probingcited by 6
  6. 2025Learning Refined Document Representations for Dense Retrieval via Deliberate Thinkingcited by 5
  7. 2025R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learningcited by 3
  8. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Modelscited by 1
  9. 2026Measuring AI Reasoning: A Guide for Researcherscited by 1
  10. 2026Mechanistic Interpretability of Chain-of-Thought Reasoning via Sequential Activation Patchingcited by 1
  11. 2026Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scorescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding Chain-of-Thought in LLMs through Information Theory
  2. 2023Measuring Faithfulness in Chain-of-Thought Reasoning
  3. 2025Chain-of-Thought Tokens are Computer Program Variables
  4. 2026The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
  5. 2025Early Stopping Chain-of-thoughts in Large Language Models
  6. 2026Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment
  7. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  8. 2024On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models