# Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning

arXiv: https://arxiv.org/abs/2606.07720 · HTML: https://mlsanity.com/paper/2606.07720/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 12 of 12)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://mlsanity.com/paper/2311.01460/) (2023)
- [Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small](https://mlsanity.com/paper/2409.04478/) (2024)
- [FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2509.13334/) (2025)
- [Efficient Post-Training Refinement of Latent Reasoning in Large Language Models](https://mlsanity.com/paper/2506.08552/) (2025)
- [Broken Chains: The Cost of Incomplete Reasoning in LLMs](https://mlsanity.com/paper/2602.14444/) (2026)
- [Amortized Latent Steering: Low-Cost Alternative to Test-Time Optimization](https://mlsanity.com/paper/2509.18116/) (2025)

## Built on by (most cited first, 1 of 1)

- [Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers](https://mlsanity.com/paper/2609.19521/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable reasoning abilities on mathematical and multi-hop planning tasks. The CoCoNuT (Chain of Continuous Thought) paradigm~citehao2024coconut extends this by enabling models to reason in latent space, exploring multiple reasoning paths simultaneously rather than committing to a single chain early on. However, we identify a limitation we term the concept bottleneck. At each reasoning pass, intermediate hidden states are overwritten, causing the model to lose critical facts computed in earlier steps as reasoning depth increases. We observe this empirically. On HotpotQA, vanilla CoCoNuT (10.4% EM) fails to improve over the CoT baseline (11.0% EM), and performance degrades with curriculum depth on GSM8K. To address this, we propose AGCLR (Adaptive Gated Continuous Latent Reasoning), which augments CoCoNuT with a textitGated Concept Stream. A persistent residual memory maintained across all reasoning passes, controlled by three learned gates: a textitwrite gate that commits intermediate facts to memory, a textitread gate that retrieves relevant prior states, and a textitforget gate that prunes irrelevant context. Evaluated on GSM8K, HotpotQA, and ProsQA using GPT-2 as our base model, AGCLR achieves consistent improvements across all types of datasets. With the performance gap compounding as curriculum depth increases, directly resolving the concept bottleneck. Code available at https://anonymous.4open.science/r/JJJJ/README.md