# Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer

arXiv: https://arxiv.org/abs/2507.02199 · HTML: https://mlsanity.com/paper/2507.02199/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2406.09136/) (2024)
- [Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought](https://mlsanity.com/paper/2410.05695/) (2024)

## Built on by (most cited first, 27 of 27)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers](https://mlsanity.com/paper/2604.07822/) (2026)
- [A Mechanistic Analysis of Looped Reasoning Language Models](https://mlsanity.com/paper/2604.11791/) (2026)
- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Training-Free Looped Transformers](https://mlsanity.com/paper/2605.23872/) (2026)
- [Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks](https://mlsanity.com/paper/2602.00449/) (2026)
- [Are Latent Reasoning Models Easily Interpretable?](https://mlsanity.com/paper/2604.04902/) (2026)
- [Loop the Loopies!](https://mlsanity.com/paper/2607.16051/) (2026)
- [When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers](https://mlsanity.com/paper/2607.20594/) (2026)
- [Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?](https://mlsanity.com/paper/2609.01924/) (2026)
- [Reducing Belief Deviation in Reinforcement Learning for Active Reasoning](https://mlsanity.com/paper/2510.12264/) (2025)
- [LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling](https://mlsanity.com/paper/2606.18023/) (2026)
- [Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Intervention](https://mlsanity.com/paper/2606.01243/) (2026)
- [Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation](https://mlsanity.com/paper/2605.30757/) (2026)
- [The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner](https://mlsanity.com/paper/2608.03263/) (2026)
- [The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA](https://mlsanity.com/paper/2609.03090/) (2026)
- [PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning](https://mlsanity.com/paper/2602.03190/) (2026)
- [Steering Recurrent Reasoners at Inference Time with Readout Feedback](https://mlsanity.com/paper/2608.24136/) (2026)
- [SMolLM: Small Language Models Learn Small Molecular Grammar](https://mlsanity.com/paper/2605.06322/) (2026)
- [LatentMT: Machine Translation with Latent Reasoning](https://mlsanity.com/paper/2607.18618/) (2026)
- [Behaviour Is an Incomplete Measure of Reasoning Development: Cross-surface pre-arrival accessibility and the limits of developmental inference in a recurrent-depth reasoner](https://mlsanity.com/paper/2608.16085/) (2026)
- [StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions](https://mlsanity.com/paper/2609.01081/) (2026)
- [Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It](https://mlsanity.com/paper/2609.36585/) (2026)
- [LoopVL: Recurrent Visual Intelligence](https://mlsanity.com/paper/2609.38426/) (2026)
- [Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does](https://mlsanity.com/paper/2609.39892/) (2026)
- [Depth as Time in One-Step Generative Models](https://mlsanity.com/paper/2610.03626/) (2026)
- [Monitoring latent reasoning in looped language models](https://mlsanity.com/paper/2610.18127/) (2026)

## Abstract (arXiv)

Chain-of-thought (CoT) reasoning has enabled transformer-based language models to excel at complex mathematics and multi-step planning. However, in standard decoder-only architectures, these reasoning steps are externalized in natural language, improving interpretability at the cost of efficiency. To capture reasoning that is not easily represented in words, many works have explored recurrent architectures that aim to internalize reasoning in latent space, potentially supporting latent CoT. In this paper, we investigate whether such reasoning structures emerge in Huginn-3.5B, a depth-recurrent Transformer that reuses layers at inference time without increasing parameter count. We examine the model's internal behavior on arithmetic tasks using a suite of probing techniques including the Logit Lens and Coda Lens. Our findings reveal limited evidence of interpretable latent CoT by tracking rank trajectories of final and intermediate result tokens. Furthermore, we uncover significant probing inconsistencies across recurrent blocks, where the interpretability of hidden states depends heavily on both the layer index and the decoding method. Finally, we empirically show that increasing recurrence depth yields only marginal gains and falls well short of models that explicitly externalize reasoning steps. The code is available at https://github.com/wenquanlu/huginn-latent-cot.