# LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics arXiv: https://arxiv.org/abs/2412.01113 ยท HTML: https://mlsanity.com/paper/2412.01113/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. - Builds on 26 of its 42 references that are in the map. - First posted on arXiv 2 Dec 2024. ## Placed in - LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/) ## Builds on (most cited first, 26 of 26) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016) - [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023) - [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024) - [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022) - [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023) - [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022) - [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024) - [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023) - [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023) - [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024) - [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019) - [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019) - [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024) - [Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process](https://mlsanity.com/paper/2407.20311/) (2024) - [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024) - [Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?](https://mlsanity.com/paper/2312.03729/) (2023) - [Iteration Head: A Mechanistic Study of Chain-of-Thought](https://mlsanity.com/paper/2406.02128/) (2024) - [Chain-of-Thought Unfaithfulness as Disguised Accuracy](https://mlsanity.com/paper/2402.14897/) (2024) - [Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching](https://mlsanity.com/paper/2311.15131/) (2023) - [Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination](https://mlsanity.com/paper/2311.09627/) (2023) - [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024) - [Do LLMs Really Think Step-by-step In Implicit Reasoning?](https://mlsanity.com/paper/2411.15862/) (2024) - [Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?](https://mlsanity.com/paper/2302.07866/) (2023) ## Built on by (most cited first, 11 of 11) - [Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning](https://mlsanity.com/paper/2505.16782/) (2025) - [Implicit Reasoning in Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.02350/) (2025) - [Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers](https://mlsanity.com/paper/2502.02393/) (2025) - [Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks](https://mlsanity.com/paper/2602.00449/) (2026) - [Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing](https://mlsanity.com/paper/2603.17199/) (2026) - [Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking](https://mlsanity.com/paper/2502.12974/) (2025) - [R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning](https://mlsanity.com/paper/2509.25987/) (2025) - [Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models](https://mlsanity.com/paper/2601.14270/) (2026) - [Measuring AI Reasoning: A Guide for Researchers](https://mlsanity.com/paper/2605.02442/) (2026) - [Mechanistic Interpretability of Chain-of-Thought Reasoning via Sequential Activation Patching](https://mlsanity.com/paper/2608.22332/) (2026) - [Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores](https://mlsanity.com/paper/2608.31068/) (2026) ## Abstract (arXiv) This study investigates the internal information flow of large language models (LLMs) while performing chain-of-thought (CoT) style reasoning. Specifically, with a particular interest in the faithfulness of the CoT explanation to LLMs' final answer, we explore (i) when the LLMs' answer is (pre)determined, especially before the CoT begins or after, and (ii) how strongly the information from CoT specifically has a causal effect on the final answer. Our experiments with controlled arithmetic tasks reveal a systematic internal reasoning mechanism of LLMs. They have not derived an answer at the moment when input was fed into the model. Instead, they compute (sub-)answers while generating the reasoning chain on the fly. Therefore, the generated reasoning chains can be regarded as faithful reflections of the model's internal computation.